Cost optimization for LLM apps

Every LLM feature ships in the same order. Quality first. Cost second. Latency third.

For 3 months the team focuses on quality — every prompt tune, every retrieval improvement, every eval win. The feature ships. Users love it. Finance sends a screenshot: "Our OpenAI bill went from $2K to $47K last month. Can we do something?"

You stare at the invoice. gpt-4o requests, hundreds of millions of them. Every user chat is 3-8K input tokens (system prompt + RAG context + history), 500-1500 output tokens. Multiply by 10M chats/month. You do the math: $47K is actually cheap for this workload. Doubling users means doubling the bill.

You have to cut cost by 5-10x, and quality can't move. This is a real engineering problem — solvable with a repeatable playbook.

Here's the playbook. Every production LLM team eventually converges on the same 5 levers. Apply them in order, and 10x savings is the typical outcome (not a lucky maximum). The teams that struggle are the ones who only try one lever and give up.

The cost math you need to internalize

Before the levers, the pricing shape. In late 2025, on the frontier models:

• Input tokens — ~$1-3 per million on flagship models (GPT-4o, Claude 3.5 Sonnet), ~$0.10-0.30 per million on cost-tier models (GPT-4o mini, Claude Haiku, Gemini Flash)
• Output tokens — ~$4-15 per million on flagship, ~$0.40-1.20 on cost-tier
• Ratio to remember — output is 3-5× the price of input. Every output token spent is 3-5 input tokens.
• Cached input tokens — ~50% discount (Anthropic, OpenAI) or ~90% discount (Google Gemini implicit cache)
• Batch API — 50% discount on both input and output, with 24h SLA (OpenAI + Anthropic)

The single most useful rule of thumb: a typical production chat is ~80% input tokens (system prompt + RAG context + history) and ~20% output tokens (the answer). Because output is priced 3-5x higher, cost is roughly ~50% driven by input volume and ~50% by output volume — both matter, and the levers that reduce each are different.

The 5 cost levers — the only diagram you need

Rule: apply the levers in order. Prompt caching is nearly free and gives the biggest immediate win. Model routing is the second biggest lever and the one most teams underestimate. Semantic caching + batching + distillation come later when volume justifies the engineering effort.

Let me walk through each.

Lever 1: Prompt caching — free money you're leaving on the table

What it is: every provider now supports caching the invariant prefix of your prompts. Your system prompt + few-shot examples + RAG context that doesn't change between requests gets stored server-side; subsequent requests reference it at a 50-90% discount.

Anthropic cache_control (typical usage):

python
    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        messages=[
            {"role": "user", "content": [
                {"type": "text", "text": "SYSTEM: long instructions..."
                                          + "RAG CONTEXT: 20K tokens...",
                 "cache_control": {"type": "ephemeral"}},
                {"type": "text", "text": user_question},  # not cached
            ]}
        ],
    )
    # First call: full price on all tokens + small write premium (~25%)
    # Subsequent calls within 5 min: cached prefix is ~10% of normal price

Openai prompt caching (automatic since Oct 2024): any prefix ≥1024 tokens that repeats across requests within ~1h is cached automatically at 50% off. You don't do anything — it just works.

Gemini implicit context caching: automatic + explicit APIs, discounts up to ~75%.

The typical win: a chat product with a 3K-token system prompt + 5K-token RAG context sees ~50% input cost reduction just by wrapping the invariant portion in a cache directive. Zero risk to quality. Ship this first, this week.

Gotchas:

• Cache TTL is short (5 min on Anthropic ephemeral, ~1h on OpenAI, longer explicit tiers exist). If your users come in bursty, hit rate is great; if uniformly distributed, worse.
• Any change to the prefix invalidates the cache — even whitespace. Version your prompts carefully.
• Write premium: first call has +25% cost on cached tokens. Only pays off if the cache is hit ≥2 times.

Lever 2: Model routing — the biggest lever nobody uses

The insight: not every query needs GPT-4o. On a typical product distribution:

• 60-80% of queries are simple — greetings, FAQ, factual lookups. Cost-tier models (GPT-4o mini, Claude Haiku, Gemini Flash) handle these fine at ~1/20th the price.
• 15-30% are medium — moderate reasoning, standard code, structured extraction. Cost-tier can do these but with quality gap.
• 5-15% are hard — long-form reasoning, complex code, tricky edge cases. These need the flagship.

If you route 80% of traffic to a 1/20th-cost model, your blended cost drops ~4-5x. That's the single biggest cost lever after caching.

text
    ══════════ MODEL ROUTING PIPELINE ══════════

    User query arrives
           ↓
    Classifier assigns tier:
      - "cheap-tier"   (60-80% of queries)
      - "flagship-tier" (5-15% of queries)
      - "hybrid" — try cheap, escalate if low confidence
                                      (15-30% of queries)
           ↓
    Cheap-tier: GPT-4o mini / Haiku / Flash
    Flagship-tier: GPT-4o / Claude Sonnet 3.5 / Gemini Pro
           ↓
    Response

    Blended cost at 70% cheap / 30% flagship:
      = 0.70 × $0.15/M + 0.30 × $2.50/M   (rough input pricing)
      = $0.86/M blended
    vs $2.50/M all-flagship = 2.9x savings on INPUT
    OUTPUT savings even higher because output prices scale worse

How to build the classifier:

• Option A — Rule-based (start here): length of query, presence of code, complexity heuristics. 2 hours of work, 5-10x speedup, ~85% correct routing.
• Option B — Embedding classifier: cluster past queries by embedding, learn per-cluster model preference from eval scores. 1 week, ~92% correct routing.
• Option C — Cascade with confidence (best): always run cheap-tier first; if the cheap-tier output has low confidence (below a threshold on judge score or reported logprobs), re-run with flagship. Zero routing errors, but 25-40% of queries pay double.
• Option D — Learned router: fine-tune a small classifier on (query → best-model) labels. 2 weeks, ~95% correct routing.

RouteLLM (Ong et al., 2024, arxiv.org/abs/2406.18665) benchmarks routing strategies — well worth reading before you build one.

Latency bonus: cheap-tier models are typically 2-3x faster. Model routing improves latency as well as cost.

Lever 3: Semantic caching — cache the answer, not just the prompt

What it is: for read-heavy patterns (Q&A, FAQ chatbots), embed the incoming query and check if a semantically-similar query was answered recently. If cosine similarity to a cached query is >0.95, return the cached answer instead of calling the LLM.

Typical hit rate: 10-30% on Q&A workloads (huge variance by product). Every hit = ~100% cost savings on that request.

Anatomy:

python
    def get_cached_or_generate(query: str) -> str:
        query_embedding = embed(query)
        hits = vector_store.similar(query_embedding, threshold=0.95, k=1)
        if hits:
            # Cache hit — return stored response, log for hit-rate metrics
            return hits[0].response
        # Cache miss — call LLM
        response = llm.chat(query)
        vector_store.insert(query_embedding, response, ttl_hours=24)
        return response

Gotchas:

• Personalized queries can't be cached — "what's MY order status?" needs the user's ID; caching across users leaks data. Scope caches by user ID or exclude personalized queries.
• Time-sensitive queries — "what's the weather?" changes every hour; short TTL.
• Similarity threshold — 0.95 is a starting point. Too low → returns wrong answers. Too high → very low hit rate. A/B test on quality metrics.
• Cache invalidation — when the underlying data (product catalog, docs) changes, cache must invalidate. Version-tag caches or accept staleness.

Tools: GPTCache, Langchain's SemanticCache, Redis Vector + custom logic, or your existing vector DB (Pinecone, Weaviate, Qdrant) with an extra collection.

Lever 4: Batch API — 50% off for async workloads

What it is: OpenAI Batch API and Anthropic Message Batches accept up to 50K requests in a single upload, process them within 24h, return results. Half price on both input and output.

When it fits:

• Nightly summarization jobs — summarize the day's support tickets, generate personalized emails, tag new documents.
• Data pipeline work — extract structured data from documents, classify customer feedback, generate embeddings-adjacent content.
• Offline eval runs — run your golden eval set through 5 model variants for a comparison; batch cuts eval cost in half.

When it doesn't fit: chat, real-time features, anything with a P99 latency budget under 24h. Which is most consumer-facing use cases.

The insight: many teams have a mix of real-time and async workloads. The async ones are often 20-40% of total token volume. Moving those to batch drops that 20-40% by half → 10-20% overall bill reduction.

Lever 5: Distillation — the endgame

What it is: collect thousands of high-quality (input, output) pairs generated by a flagship model. Fine-tune a much smaller model on those pairs. Deploy the small model in production. It doesn't need to match the teacher exactly — it just needs to be good enough for your task-specific distribution.

Typical savings: 10-50x on token pricing. A fine-tuned Llama 3.1 8B on your workload runs ~$0.20/M tokens (self-hosted) vs $2.50/M for Claude Sonnet.

Cost of the effort:

• Data collection: 5K-50K teacher-generated examples with acceptance filtering. ~$500-$5K at flagship prices.
• Fine-tuning: hosted (OpenAI, Together, Anyscale) $50-$500 per run. Self-hosted a Llama on Modal or Replicate: ~$100-$1K per run.
• Serving: hosted small model APIs or self-hosted GPU pool. Break-even is typically ~1M requests/month.

When it fits:

• Volume high enough for infrastructure amortization to matter (~1M+ requests/month).
• Task narrow enough that the small model can learn it well (classification, extraction, template following, style matching — YES; open-domain reasoning, code — HARDER).
• You have evals that catch quality regressions.

When it doesn't fit: low volume, wide-open task, no eval discipline. The engineering + ops cost dominates.

The end-to-end pattern — how a team applies all 5

The compounding effect: Lever 1 (prompt caching) cuts input cost 50%. Lever 2 (model routing 70/30 split) cuts blended cost another 3x. Lever 3 (semantic cache, 20% hit rate) removes 20% of remaining calls entirely. Lever 4 (batching offline workloads) halves 30% of the remaining. Result: 47 × 0.5 × 0.33 × 0.8 × 0.85 = $5.3K/mo instead of $47K/mo. Real numbers, from a real team.

The L4 → L7 cost-engineering ladder

• L4 (starter): flagship-only, no caching. Fine while volume is low. Cost per user typically $0.10-$1.00/mo.
• L5 (production): prompt caching + basic model routing (2 tiers). Standard for anything with meaningful volume. Cost per user typically $0.02-$0.20/mo.
• L6 (advanced): all 5 levers, per-user cost dashboard, alerting on cost-per-conversation regressions, monthly cost reviews as part of release process. Cost per user typically $0.005-$0.05/mo.
• L7 (frontier): dedicated cost-engineering team, custom serving stack for distilled models, per-tenant cost budgets, real-time cost prediction per request, learned routers with reinforcement learning. Perplexity, Anthropic itself, Cursor, Google AI all operate here.

The most common mistakes

1. "We'll fix cost later." By month 6 with 5x more users you're paying 5x more. Baseline cost engineering (Levers 1 + basic Lever 2) should ship with the feature, not after.

2. "GPT-4o mini is bad, so we can't route." Test it on YOUR queries — mini has closed most of the gap on many tasks. Route by task, not by feeling.

3. "Semantic cache breaks personalization." Scope caches per user or exclude personalized queries. Non-personalized FAQ answers are almost always safe to cache.

4. "Batching is too complex." Batch APIs are literally 20 lines. If any part of your workload is async, use them.

5. "Distillation is the answer to everything." No. It's the endgame after you exhaust the easier levers. If you distill at 100K QPM you save meaningfully. At 10K QPM the engineering cost dominates.

6. "Cost dashboards are Finance's problem." Every LLM engineer should see cost-per-user, cost-per-feature, and cost-per-model version on their dashboard. Without this, cost regressions ship silently.

What's next in this journey:

• Chapter 1: Prompt caching in depth — Anthropic ephemeral vs 1h vs 5m tiers, OpenAI automatic caching mechanics, cache-key design patterns
• Chapter 2: Building a model router — rule-based → embedding classifier → cascade with confidence → learned router (RouteLLM patterns)
• Chapter 3: Semantic caching with GPTCache + designing invalidation strategies
• Chapter 4: Batch API integration and identifying async-safe workloads in your product
• Chapter 5: Distillation end-to-end — data collection, fine-tuning with OpenAI/Together, deployment, quality gates
• Chapter 6: Cost dashboards + FinOps for LLMs — the metrics + alerting a mature team runs

Sources cited in this chapter:

• OpenAI pricing: openai.com/api/pricing
• Anthropic pricing + prompt caching: docs.anthropic.com/claude/docs/prompt-caching
• Google Gemini pricing + context caching: ai.google.dev/gemini-api/docs/caching
• OpenAI Batch API: platform.openai.com/docs/guides/batch
• Anthropic Message Batches: docs.anthropic.com/claude/docs/message-batches
• RouteLLM (Ong et al., 2024): arxiv.org/abs/2406.18665
• GPTCache: github.com/zilliztech/GPTCache
• LangChain SemanticCache: python.langchain.com/docs/how_to/llm_caching
• Chip Huyen — "AI cost engineering": huyenchip.com/2024/01/16/ai-costs.html
• The distillation-for-production pattern: openai.com/research/distillation