Prompt optimizer
Writes, repairs or ports system and agent prompts against a small eval set, comparing candidate rewrites on the same cases and checking the winner on held-out ones. Use when a prompt is unreliable, moves to another model family or needs evals.
prompt-optimizer- Author
- Sentry
- Source
- getsentry/skills/skills/prompt-optimizer @ a582c51
- License
- Apache-2.0 · Copyright 2025 Functional Software, Inc. dba Sentry
About this skill
What it does
The skill writes, rewrites or ports a prompt — system, agent or template prompts and tool descriptions — and judges each change on a small eval set rather than by eye. First it records the contract: target model, prompt layers and owners, goal, output shape, success criteria, failure cases, hard limits. Without success criteria or examples it builds an eval set. If the real problem is the model, retrieval, a tool schema or missing evals, it says so before rewriting.
Governing repository files such as AGENTS.md are referenced by exact path, not pasted. Stable policy goes in system or developer messages, task data in user sections, one owner per rule. It never asks the model for chain-of-thought.
Optimizing is a loop: baseline on a fixed slice, group failures by cause, write concrete edit critiques, build two to four candidates (minimal repair, restructure, example-led variant, provider adapter), compare them on the same cases, log each round, check the winner on held-out cases, and stop on plateau or overfitting. The output has eight parts: target, success criteria, external context, the prompt, adapter notes, eval set, optimization log, residual risks.
References load per task: core-patterns.md (layering, markers, a symptom-to-fix table), meta-optimization-loop.md (the loop, five to ten cases plus a holdout), model-family-notes.md (OpenAI, Claude, Gemini) and transformed-examples.md (three before/after prompts). SOURCES.md lists the docs and papers behind it, SPEC.md the maintainers' contract. No scripts ship; running candidates is up to the agent's own tools.
When to use it, and when not
Use it when a production prompt is unstable, moves to another model family, or will be reused by others.
It is heavy for a one-off question. Writing skills and tuning only parameters are outside its scope.
How to run it
The install command below puts the folder in place. The agent picks the skill up by its description; to call it yourself:
- Claude Code:
/prompt-optimizer, or ask for the skill by name. - Codex:
$prompt-optimizer. - Cursor: type
/in the Agent chat and pickprompt-optimizer.
Example: /prompt-optimizer Our support-bot prompt ignores the refund policy in one reply out of five. Here are 8 transcripts; fix it.
How it differs from similar skills
LLM cost optimizer from our catalog also trims prompts, but to cut the bill. Model picker covers the case this skill only flags: the model itself. Writing for agents and Skill writer by the Agent Skills spec cover skills and AGENTS.md files.
Install
Run one command — it saves SKILL.md and the files it uses into your agent's skills folder. Claude Code and Codex pick up a new skill without a restart; if it does not show up, restart the agent.
curl -fsSL --create-dirs -o ~/.claude/skills/prompt-optimizer/SKILL.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md -o ~/.claude/skills/prompt-optimizer/LICENSE https://anymodel.org/en/skills/sentry/prompt-optimizer/LICENSE -o ~/.claude/skills/prompt-optimizer/SOURCES.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SOURCES.md -o ~/.claude/skills/prompt-optimizer/SPEC.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SPEC.md -o ~/.claude/skills/prompt-optimizer/references/core-patterns.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/core-patterns.md -o ~/.claude/skills/prompt-optimizer/references/meta-optimization-loop.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/meta-optimization-loop.md -o ~/.claude/skills/prompt-optimizer/references/model-family-notes.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/model-family-notes.md -o ~/.claude/skills/prompt-optimizer/references/transformed-examples.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/transformed-examples.md
curl -fsSL --create-dirs -o ~/.agents/skills/prompt-optimizer/SKILL.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md -o ~/.agents/skills/prompt-optimizer/LICENSE https://anymodel.org/en/skills/sentry/prompt-optimizer/LICENSE -o ~/.agents/skills/prompt-optimizer/SOURCES.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SOURCES.md -o ~/.agents/skills/prompt-optimizer/SPEC.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SPEC.md -o ~/.agents/skills/prompt-optimizer/references/core-patterns.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/core-patterns.md -o ~/.agents/skills/prompt-optimizer/references/meta-optimization-loop.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/meta-optimization-loop.md -o ~/.agents/skills/prompt-optimizer/references/model-family-notes.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/model-family-notes.md -o ~/.agents/skills/prompt-optimizer/references/transformed-examples.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/transformed-examples.md
foreach ($f in 'SKILL.md','LICENSE','SOURCES.md','SPEC.md','references/core-patterns.md','references/meta-optimization-loop.md','references/model-family-notes.md','references/transformed-examples.md') { $p = Join-Path "$HOME\.claude\skills\prompt-optimizer" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/sentry/prompt-optimizer/$f" -UseBasicParsing -OutFile $p }foreach ($f in 'SKILL.md','LICENSE','SOURCES.md','SPEC.md','references/core-patterns.md','references/meta-optimization-loop.md','references/model-family-notes.md','references/transformed-examples.md') { $p = Join-Path "$HOME\.agents\skills\prompt-optimizer" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/sentry/prompt-optimizer/$f" -UseBasicParsing -OutFile $p }Any other agent: save the file into its skills folder, or give it the link to the file: https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md
SKILL.md
Prompt Optimizer
Optimize prompts with evals. Keep every instruction, example, and external context reference causal.
Load Only What You Need
| Need | Read |
|---|---|
| New prompt | references/core-patterns.md, references/model-family-notes.md, references/transformed-examples.md |
| Existing prompt | references/meta-optimization-loop.md, references/core-patterns.md, references/model-family-notes.md |
| Model-family port | references/model-family-notes.md, references/core-patterns.md |
| Repeated failures | references/meta-optimization-loop.md, references/core-patterns.md |
| Weak or ambiguous draft | references/transformed-examples.md |
| Provenance | SOURCES.md |
Step 1: Capture Contract
Record before editing:
- task type: new, refine, port, or debug
- target model family and snapshot, if known
- prompt surface:
system,developer,user, tool descriptions, examples, schemas - layer owners: platform, deployer/persona, retrieved context, user payload
- objective and non-goals
- inputs, tools, and external files available
- required output shape
- success criteria and failure cases
- hard constraints: latency, verbosity, safety, budget, tool use, style
If success criteria or examples are missing, create a small eval set first. If the bottleneck is model choice, retrieval, tool schema, or missing evals, say so before rewriting.
Step 2: Inventory External Context
For repo or agent prompts, list stable context by exact path:
| Context type | Examples |
|---|---|
| Agent rules | AGENTS.md, CLAUDE.md |
| Specs | specs/*.md, docs/api.md |
| Policies | SECURITY.md, docs/releasing.md |
| Examples | examples/, tests/fixtures/ |
Rules:
- Reference stable files by repo-relative path instead of copying them.
- Paste only excerpts needed for the prompt or eval case.
- Mark whether a file is
loaded,referenced, orout of scope. - Avoid vague context pointers such as "read the docs".
Step 3: Choose Model Strategy
Read references/model-family-notes.md.
- Known family: optimize for that family.
- Unknown family: write a portable base plus short adapter notes.
- Snapshot changes: rerun evals.
- Cross-family divergence: specialize only the failing layer.
Step 4: Shape Prompt
Read references/core-patterns.md.
- Put stable policy in
systemordeveloper. - Put task-local facts, retrieved context, and variables in user-facing sections.
- Keep one owner per behavior rule.
- Use headings or tags only to separate content types.
- Put tool policy in prompt text; keep schemas in provider-native tools.
- Keep persona light unless it changes behavior.
- Use the shortest wording that preserves the constraint.
- Cut filler, repeated reminders, dead examples, and rationale that does not affect evals.
Step 5: Optimize
Read references/meta-optimization-loop.md for refinements.
- Baseline the current prompt on the same eval slice.
- Cluster failures by root cause.
- Write concrete edit criticisms.
- Generate two to four candidates:
- minimal-diff repair
- structure-first rewrite
- examples-first or tool-rule variant
- provider adapter when needed
- Compare candidates on the same cases.
- Keep a short optimization log.
- Validate the winner on holdout cases.
- Stop on plateau, oscillation, overfit, excessive cost, or non-prompt bottleneck.
Step 6: Return Package
Return:
TargetSuccess CriteriaExternal ContextOptimized PromptAdapter NotesEval SetOptimization LogResidual Risks
For existing prompts, include a concise diff-style note of the main behavioral changes.
Failure Modes
- editing before defining the eval target
- mixing policy, examples, and raw context without boundaries
- duplicating rules across layers
- putting durable policy in user payloads
- asking for chain-of-thought
- keeping contradictory legacy instructions
- overfitting to one or two examples
- retaining examples that no longer improve evals
- fixing tool-use failures only in prompt text when tool descriptions or schemas are weak
- adding markup that does not reduce ambiguity
- using persona as a substitute for behavior rules
Connect through AnyModel API
This skill runs on whatever model your agent uses. Point Claude Code, Codex or Cursor at AnyModel to run it on Claude, GPT, Gemini and more with one key — pay per token, no subscription.
AnyModel