Prompt optimizer

Writes, repairs or ports system and agent prompts against a small eval set, comparing candidate rewrites on the same cases and checking the winner on held-out ones. Use when a prompt is unreliable, moves to another model family or needs evals.

AI models & APIsPack: Sentry skills122 linesprompt-optimizer
Author
Source
getsentry/skills/skills/prompt-optimizer @ a582c51
License
Apache-2.0 · Copyright 2025 Functional Software, Inc. dba Sentry

About this skill

What it does

The skill writes, rewrites or ports a prompt — system, agent or template prompts and tool descriptions — and judges each change on a small eval set rather than by eye. First it records the contract: target model, prompt layers and owners, goal, output shape, success criteria, failure cases, hard limits. Without success criteria or examples it builds an eval set. If the real problem is the model, retrieval, a tool schema or missing evals, it says so before rewriting.

Governing repository files such as AGENTS.md are referenced by exact path, not pasted. Stable policy goes in system or developer messages, task data in user sections, one owner per rule. It never asks the model for chain-of-thought.

Optimizing is a loop: baseline on a fixed slice, group failures by cause, write concrete edit critiques, build two to four candidates (minimal repair, restructure, example-led variant, provider adapter), compare them on the same cases, log each round, check the winner on held-out cases, and stop on plateau or overfitting. The output has eight parts: target, success criteria, external context, the prompt, adapter notes, eval set, optimization log, residual risks.

References load per task: core-patterns.md (layering, markers, a symptom-to-fix table), meta-optimization-loop.md (the loop, five to ten cases plus a holdout), model-family-notes.md (OpenAI, Claude, Gemini) and transformed-examples.md (three before/after prompts). SOURCES.md lists the docs and papers behind it, SPEC.md the maintainers' contract. No scripts ship; running candidates is up to the agent's own tools.

When to use it, and when not

Use it when a production prompt is unstable, moves to another model family, or will be reused by others.

It is heavy for a one-off question. Writing skills and tuning only parameters are outside its scope.

How to run it

The install command below puts the folder in place. The agent picks the skill up by its description; to call it yourself:

  • Claude Code: /prompt-optimizer, or ask for the skill by name.
  • Codex: $prompt-optimizer.
  • Cursor: type / in the Agent chat and pick prompt-optimizer.

Example: /prompt-optimizer Our support-bot prompt ignores the refund policy in one reply out of five. Here are 8 transcripts; fix it.

How it differs from similar skills

LLM cost optimizer from our catalog also trims prompts, but to cut the bill. Model picker covers the case this skill only flags: the model itself. Writing for agents and Skill writer by the Agent Skills spec cover skills and AGENTS.md files.

Install

Run one command — it saves SKILL.md and the files it uses into your agent's skills folder. Claude Code and Codex pick up a new skill without a restart; if it does not show up, restart the agent.

Claude Code — macOS, Linux
curl -fsSL --create-dirs -o ~/.claude/skills/prompt-optimizer/SKILL.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md -o ~/.claude/skills/prompt-optimizer/LICENSE https://anymodel.org/en/skills/sentry/prompt-optimizer/LICENSE -o ~/.claude/skills/prompt-optimizer/SOURCES.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SOURCES.md -o ~/.claude/skills/prompt-optimizer/SPEC.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SPEC.md -o ~/.claude/skills/prompt-optimizer/references/core-patterns.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/core-patterns.md -o ~/.claude/skills/prompt-optimizer/references/meta-optimization-loop.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/meta-optimization-loop.md -o ~/.claude/skills/prompt-optimizer/references/model-family-notes.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/model-family-notes.md -o ~/.claude/skills/prompt-optimizer/references/transformed-examples.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/transformed-examples.md
Codex, Cursor — macOS, Linux
curl -fsSL --create-dirs -o ~/.agents/skills/prompt-optimizer/SKILL.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md -o ~/.agents/skills/prompt-optimizer/LICENSE https://anymodel.org/en/skills/sentry/prompt-optimizer/LICENSE -o ~/.agents/skills/prompt-optimizer/SOURCES.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SOURCES.md -o ~/.agents/skills/prompt-optimizer/SPEC.md https://anymodel.org/en/skills/sentry/prompt-optimizer/SPEC.md -o ~/.agents/skills/prompt-optimizer/references/core-patterns.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/core-patterns.md -o ~/.agents/skills/prompt-optimizer/references/meta-optimization-loop.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/meta-optimization-loop.md -o ~/.agents/skills/prompt-optimizer/references/model-family-notes.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/model-family-notes.md -o ~/.agents/skills/prompt-optimizer/references/transformed-examples.md https://anymodel.org/en/skills/sentry/prompt-optimizer/references/transformed-examples.md
Claude Code — Windows PowerShell
foreach ($f in 'SKILL.md','LICENSE','SOURCES.md','SPEC.md','references/core-patterns.md','references/meta-optimization-loop.md','references/model-family-notes.md','references/transformed-examples.md') { $p = Join-Path "$HOME\.claude\skills\prompt-optimizer" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/sentry/prompt-optimizer/$f" -UseBasicParsing -OutFile $p }
Codex, Cursor — Windows PowerShell
foreach ($f in 'SKILL.md','LICENSE','SOURCES.md','SPEC.md','references/core-patterns.md','references/meta-optimization-loop.md','references/model-family-notes.md','references/transformed-examples.md') { $p = Join-Path "$HOME\.agents\skills\prompt-optimizer" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/sentry/prompt-optimizer/$f" -UseBasicParsing -OutFile $p }

Any other agent: save the file into its skills folder, or give it the link to the file: https://anymodel.org/en/skills/sentry/prompt-optimizer/SKILL.md

SKILL.md

Raw

Prompt Optimizer

Optimize prompts with evals. Keep every instruction, example, and external context reference causal.

Load Only What You Need

Need Read
New prompt references/core-patterns.md, references/model-family-notes.md, references/transformed-examples.md
Existing prompt references/meta-optimization-loop.md, references/core-patterns.md, references/model-family-notes.md
Model-family port references/model-family-notes.md, references/core-patterns.md
Repeated failures references/meta-optimization-loop.md, references/core-patterns.md
Weak or ambiguous draft references/transformed-examples.md
Provenance SOURCES.md

Step 1: Capture Contract

Record before editing:

  • task type: new, refine, port, or debug
  • target model family and snapshot, if known
  • prompt surface: system, developer, user, tool descriptions, examples, schemas
  • layer owners: platform, deployer/persona, retrieved context, user payload
  • objective and non-goals
  • inputs, tools, and external files available
  • required output shape
  • success criteria and failure cases
  • hard constraints: latency, verbosity, safety, budget, tool use, style

If success criteria or examples are missing, create a small eval set first. If the bottleneck is model choice, retrieval, tool schema, or missing evals, say so before rewriting.

Step 2: Inventory External Context

For repo or agent prompts, list stable context by exact path:

Context type Examples
Agent rules AGENTS.md, CLAUDE.md
Specs specs/*.md, docs/api.md
Policies SECURITY.md, docs/releasing.md
Examples examples/, tests/fixtures/

Rules:

  • Reference stable files by repo-relative path instead of copying them.
  • Paste only excerpts needed for the prompt or eval case.
  • Mark whether a file is loaded, referenced, or out of scope.
  • Avoid vague context pointers such as "read the docs".

Step 3: Choose Model Strategy

Read references/model-family-notes.md.

  • Known family: optimize for that family.
  • Unknown family: write a portable base plus short adapter notes.
  • Snapshot changes: rerun evals.
  • Cross-family divergence: specialize only the failing layer.

Step 4: Shape Prompt

Read references/core-patterns.md.

  • Put stable policy in system or developer.
  • Put task-local facts, retrieved context, and variables in user-facing sections.
  • Keep one owner per behavior rule.
  • Use headings or tags only to separate content types.
  • Put tool policy in prompt text; keep schemas in provider-native tools.
  • Keep persona light unless it changes behavior.
  • Use the shortest wording that preserves the constraint.
  • Cut filler, repeated reminders, dead examples, and rationale that does not affect evals.

Step 5: Optimize

Read references/meta-optimization-loop.md for refinements.

  1. Baseline the current prompt on the same eval slice.
  2. Cluster failures by root cause.
  3. Write concrete edit criticisms.
  4. Generate two to four candidates:
    • minimal-diff repair
    • structure-first rewrite
    • examples-first or tool-rule variant
    • provider adapter when needed
  5. Compare candidates on the same cases.
  6. Keep a short optimization log.
  7. Validate the winner on holdout cases.
  8. Stop on plateau, oscillation, overfit, excessive cost, or non-prompt bottleneck.

Step 6: Return Package

Return:

  1. Target
  2. Success Criteria
  3. External Context
  4. Optimized Prompt
  5. Adapter Notes
  6. Eval Set
  7. Optimization Log
  8. Residual Risks

For existing prompts, include a concise diff-style note of the main behavioral changes.

Failure Modes

  • editing before defining the eval target
  • mixing policy, examples, and raw context without boundaries
  • duplicating rules across layers
  • putting durable policy in user payloads
  • asking for chain-of-thought
  • keeping contradictory legacy instructions
  • overfitting to one or two examples
  • retaining examples that no longer improve evals
  • fixing tool-use failures only in prompt text when tool descriptions or schemas are weak
  • adding markup that does not reduce ambiguity
  • using persona as a substitute for behavior rules

Connect through AnyModel API

This skill runs on whatever model your agent uses. Point Claude Code, Codex or Cursor at AnyModel to run it on Claude, GPT, Gemini and more with one key — pay per token, no subscription.

More from the pack Sentry skills

AGENTS.md and CLAUDE.md writer

Writes or trims a repository's AGENTS.md to under 60 lines of commands, conventions and pointers to existing docs, checks every path, and makes CLAUDE.md a symlink to it. Use when a repo needs agent instructions or its current file has grown long or stale.

Claude Code settings audit

Detects a repository's stack and proposes a Claude Code settings.json allow list of read-only commands for the tools it actually uses, plus docs domains and Sentry or Linear MCP entries. Use when setting up Claude Code permissions or auditing existing ones.

Code review, Sentry practices

Reviews a change by Sentry's guidelines: runtime errors, performance and N+1 queries, side effects, compatibility, security, design and tests, with a polite tone and approval once only minor issues remain. Use when reviewing a pull request or branch.

Code simplifier

Simplifies recently changed code without changing behaviour: less nesting, no nested ternaries, clearer names, no redundant abstractions, while following CLAUDE.md conventions. Use when code works but reads heavier than it should and needs a cleanup pass.

Conventional commit messages

Commits in Sentry's Conventional Commits style: a feature branch instead of main, a typed capitalised subject up to 70 characters, Fixes or Refs footers, no personal data. Use when asked to commit changes or draft a commit message.

Django access control review

Audits Django and DRF code for IDOR and broken access control by learning the app's authorization model and tracing each endpoint from ID to query, reporting only confirmed gaps with enforcing fixes. Use when reviewing views, viewsets or tenant isolation.

Find bugs in branch changes

Reviews the commits on the current branch for security holes, bugs and quality issues with an eleven-item checklist and reports verified findings with file, line, severity and fix, without editing code. Use when you want a security-first pass before a PR.

GitHub Actions security review

Audits GitHub Actions workflows for attacks an outsider without write access can run: pwn requests, expression injection, comment commands, token theft, poisoned agent configs, unpinned actions. Use when reviewing CI security or changing workflows.

Iterate a PR until CI is green

Drives an open PR to green: fixes failing CI and high and medium review comments, verifies locally, commits, pushes and repeats, stopping at human approval or draft gates. Use when a PR is red or has review feedback to work through.

PR title and description writer

Writes or refreshes a PR title in Sentry's conventional format and a short reviewer-facing description without template headings or test logs, then opens a draft or updates the PR. Use when opening a PR or after commits change its scope.

Secret leaks through serialization

Finds secrets that leak when a credential field on a dataclass, pydantic model or JS object meets code that logs or traces whole objects, searching the whole repository for the other half. Use when a change adds such a field or a wholesale sink.

SECURITY_MODEL.md for review tools

Writes and maintains SECURITY_MODEL.md, telling security review tools a repository's exposure, invariants, reportable finding classes and owner-confirmed exclusions, with a script that flags stale references. Use when preparing a repo for automated review.

Agent skill security scanner

Audits an agent skill before installation: a static scan for prompt injection, hidden Unicode, secrets, risky scripts and structural traps, then a review of intent and permissions ending in a verdict. Use when a skill comes from an outside source.

Skill writer by the Agent Skills spec

Creates and improves agent skills by Sentry's workflow: sources with provenance, the simplest execution shape, SPEC.md and SOURCES.md, precision passes after each change and a structural validator. Use when building a skill that a team will maintain.

Security review, Sentry

Reviews a file, diff or component for exploitable vulnerabilities with OWASP-based references, researches the codebase before flagging and reports only findings reached by attacker-controlled input. Use when a change needs a low-noise security review.

Django performance review

Reviews Django and DRF code for N+1 queries, unbounded querysets, missing indexes and write loops, validating table size and hot paths before reporting. Use when a Django view or endpoint is slow or a change touches large tables.