---
name: llm-cost-optimizer
description: Finds where an application spends money on LLM calls and cuts it with model routing, prompt trimming, caching, batching and output limits, measuring cost and quality before and after. Use when the user says the AI bill is too high or wants to make LLM usage cheaper.
---

# LLM Cost Optimizer

Cost work starts with measurement. Cut the biggest line item first and check that quality did not
drop. Never "optimize" by guessing.

## 1. Measure where the money goes

1. Find every place the code calls a model (grep for the SDK client, `chat.completions`,
   `messages.create`, `/v1/`). List them with the model id each one uses.
2. For each call site get, from logs, the provider dashboard or a short instrumented run:
   requests per day, average input tokens, average output tokens, model price.
3. Compute daily cost per call site and sort descending. Usually one or two call sites are most
   of the bill — work only on those.

If the code does not log token usage, add it first: non-streaming responses return a `usage` object;
for OpenAI-format streaming pass `stream_options: {"include_usage": true}` to get usage in the last
chunk. Log model id, input tokens, output tokens and the call site name — never the prompt contents if they
may hold user data.

## 2. Apply the levers, biggest first

**Right-size the model.** Classification, extraction, routing and short rewrites rarely need a
flagship. Test a smaller model on 20–30 real examples (see the model-picker skill) and switch
the call sites where it passes.

**Trim the input.**
- Remove repeated instructions, examples nobody needs, and whole files where a function would do.
- Send only the relevant chunks for RAG instead of the full document.
- Cap conversation history: keep the last N turns plus a running summary.

**Cache.**
- Through AnyModel, prompt caching does not lower the bill: re-sent context is charged in full.
  A stable prefix (system prompt, tool definitions, reference docs) kept first and identical
  byte-for-byte can still make answers faster, but count savings only from the other levers.
  On a direct provider account, check that provider's caching price before relying on it.
- Cache whole responses for identical requests (hash of model + prompt + params) when the answer
  does not need to be fresh.

**Limit the output.** Set `max_tokens` to what the feature needs, ask for terse formats (JSON with
fixed keys, no preamble), and stop streaming once you have what you need.

**Batch and deduplicate.** Group many small independent items into one request where quality
holds, and skip calls whose input did not change since the last run.

**Stop runaway loops.** Agent loops and retries can multiply cost: cap iterations and retries,
and alert on a single request exceeding a token budget.

## 3. Verify

1. Re-run the same examples used before the change and compare quality: pass rate, or a diff of
   outputs reviewed by the user for subjective tasks.
2. Recompute the daily cost table with the new numbers.
3. Keep every change behind configuration (model id, max tokens, cache on/off) so it can be
   reverted without a deploy.

## Report back

```
Before: $<X>/day — top call sites: <a> $.., <b> $..
Changes: <lever> on <call site> (…)
After:  $<Y>/day (−<N>%) — quality: <pass rate before → after>
Not changed and why: <…>
```

## Pitfalls

- Optimizing a call site that is 2% of the bill.
- Switching to a cheaper model without testing on real inputs.
- Promising savings from prompt caching through AnyModel: the cached prefix is billed in full.
- Logging full prompts or responses that contain personal data while measuring.

Source: [AnyModel skills catalog](https://anymodel.org/en/skills) · License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
