Fine-tuning vs RAG — the decision

A CTO stops by your desk. "The LLM demo was great. Now the sales team wants the assistant to know our product catalog, our pricing policy, our 5,000-page compliance handbook, and our support history. What do we do — RAG or fine-tuning? The blog posts contradict each other."

You know the individual techniques. You built a RAG in A3. You've heard about fine-tuning. But this question — which one, when, why — is where every LLM team gets stuck. Some teams pick wrong and spend 6 months + $200K on fine-tuning when RAG would have shipped in 2 weeks. Other teams stay on RAG when their task needs fine-tuning and never break through a quality ceiling.

This is the single most consequential architectural decision in an AI product. It affects cost, latency, quality, deployment complexity, and how you evolve the system over time.

Once you understand the 4-question decision framework below, you can defend the choice with math, not intuition. And you'll know when the honest answer is "both" — the pattern most production systems eventually converge on.

The distinction that clears up 90% of the confusion

RAG and fine-tuning solve different classes of problem. They aren't alternatives to each other — they're alternatives for different bottlenecks.

• RAG solves the knowledge problem: how does the model access information it wasn't trained on?
• Fine-tuning solves the behavior problem: how does the model consistently produce output in a specific format, tone, or style, or make consistent decisions on ambiguous inputs?

If your problem is "the model doesn't KNOW something" → RAG.
If your problem is "the model doesn't DO something the way we want" → fine-tuning.

If your problem is both → both. Which is common.

The 4-question decision framework — the only diagram you need

Rule: Question 1 handles ~70% of decisions (most real problems have dynamic or large knowledge — RAG wins). Questions 2-4 handle the remaining 30%. When you hit hybrid, you've reached the endgame — expect to be there eventually.

Let me walk through each option in depth.

Option A: RAG — the default for knowledge problems

When it wins:

• Dynamic knowledge — pricing changes weekly; policies get updated; product catalog evolves. Fine-tuning can't keep up.
• Large knowledge — 100K+ documents. Fine-tuning would require training on gigabytes of text; RAG retrieves the relevant chunk at query time for pennies.
• Attribution required — RAG returns the source document IDs so the user (or regulator) can verify. Fine-tuning bakes the knowledge into weights with no citations.
• Domain-agnostic model choice — the same RAG pipeline works whether you're on GPT-4o, Claude, Gemini. You aren't tied to a specific model's fine-tuning story.

When it fails:

• The model doesn't know how to USE the retrieved context correctly (bad format, wrong tone, misinterprets constraints).
• Retrieval quality is the ceiling. If the right chunk isn't in the top-K, the model has no answer.
• Very long context windows (100K+ tokens of retrieved context) get expensive to run every request.

Cost profile:

• Setup: 1-2 weeks eng effort. ~$0 initial infra beyond embeddings.
• Per-request: pay for embeddings + retrieval + LLM call. Typical redirect ~$0.001-0.02 per query.
• Ongoing: index rebuilds as data changes (cheap — embeddings are commoditized).

Option B: Fine-tuning — the answer for behavior problems

When it wins:

• Consistent style/tone — you need every output to sound like your brand voice. RAG can't enforce this reliably.
• Structured output that's unusual — your task's format is niche enough that base models keep getting it wrong even with few-shot examples.
• Domain-specific reasoning — medical diagnostics, legal analysis, code in a proprietary DSL. The model needs to have INTERNALIZED the pattern.
• Latency or cost critical — a fine-tuned 8B model at $0.15/M tokens crushes a $2.50/M frontier model for a specific task.
• You have thousands of good examples — either from historical data (support tickets with correct resolutions) or generated by a bigger model + human review.

When it fails:

• Base knowledge is dynamic — you'd retrain every week. Untenable.
• You don't have enough examples (rule of thumb: 1,000+ good pairs minimum for LoRA, 10,000+ for full fine-tune).
• Task is broad — fine-tuning helps at specific tasks but doesn't preserve general capabilities well. A model fine-tuned on customer support may get worse at coding.

Cost profile:

• Setup: 4-8 weeks eng effort (data collection is usually the bottleneck).
• Training run: ~$100-$5,000 depending on model size + technique (LoRA is cheap, full fine-tune is expensive).
• Per-request: significantly cheaper long-term. A fine-tuned Llama 3.1 8B at ~$0.20/M tokens vs $2.50/M for Claude Sonnet = 12× cheaper.
• Ongoing: retrain quarterly or when model drifts. Data pipeline is the real cost.

Techniques (ordered from cheap → expensive):

• Prompt engineering — you didn't need fine-tuning. Try prompt patterns first (few-shot, CoT, structured output).
• LoRA (Low-Rank Adaptation) — train ~1% of parameters. Fast, cheap, 90% of the quality of full FT. The default fine-tuning technique in 2025.
• QLoRA — quantized LoRA. Even cheaper. Runs on consumer GPUs.
• Full fine-tuning — all parameters updated. Best quality on narrow tasks, most expensive, most fragile.
• DPO / RLHF — align to human preferences. When you have preference pairs (A > B) rather than input/output examples.

Option C: Hybrid — RAG + fine-tuning, the endgame pattern

The insight: knowledge and behavior are separable. You can have both.

• RAG layer provides current, up-to-date domain knowledge (product catalog, policies, docs).
• Fine-tuned model knows HOW to interpret and respond to that knowledge in your specific format/tone/decision style.

Anatomy:

text
    ══════════ HYBRID PATTERN ══════════

    User query
       ↓
    Retrieval (Stage 1 dense + Stage 2 rerank)
       ↓
    Top-5 chunks + query passed to LLM
       ↓
    LLM is FINE-TUNED on your task specifically:
      - Knows how to weight sources
      - Knows your response format
      - Knows your escalation policy
      - Knows domain-specific reasoning patterns
       ↓
    Response returned in your voice

Real production example (paraphrased from Cursor, Perplexity, GitHub Copilot):

• Retrieval finds the relevant code files / web pages / docs.
• A fine-tuned model rewrites them into an answer in the product's specific style + citation format.
• Cost: 3-5× RAG-only, but quality lift is 20-40% on eval sets.

When it's overkill: you don't have volume yet. Fine-tuning cost dominates until you're at 1M+ requests/month.

The cost math — when does fine-tuning pay off?

The break-even analysis:

text
    ══════════ COST COMPARISON ══════════

    Option A — RAG on Claude Sonnet 3.5
      Per request: ~$0.005 (avg 2K input + 500 output tokens)
      Setup: 2 weeks eng ≈ $30K
      Break-even: N/A (linear)

    Option B — Fine-tuned Llama 3.1 8B
      Per request: ~$0.0002 (avg same tokens, self-hosted or Together AI)
      Setup:
        - Data collection: $20K (label 5K examples)
        - Fine-tune: $500 (LoRA on Together)
        - Engineering: 4 weeks eng ≈ $60K
        = $80.5K
      Break-even vs Option A at request volume:
        = $80,500 / ($0.005 - $0.0002)
        = ~16.8M requests

    So — fine-tuning pays off if you'll do 16M+ requests
    over the lifetime of this project.
    At 100K requests/day (3M/month) that's ~5-6 months.

Below 16M requests, RAG is cheaper because the fine-tuning setup cost is amortized poorly. Above 16M requests, fine-tuning wins on per-request price.

Add a 30% buffer for retraining every 6 months and the break-even climbs to ~25M requests.

The evolution ladder — how a product moves through these choices

Most teams stay at V2 or V3 for a long time. V4 and V5 unlock when volume + quality requirements justify the engineering cost. Don't skip ahead.

The 8 most-common mistakes

1. Fine-tuning because "it sounds more sophisticated." RAG is simpler AND handles more use cases. Default to RAG. Fine-tune only after you have EVIDENCE that RAG can't clear your quality bar.

2. Fine-tuning without 1000+ high-quality examples. Result: overfitted model that performs worse than the base. If you don't have data, RAG. Or invest in data first.

3. Using RAG when the problem is behavior, not knowledge. "The output is in the wrong format" is not fixed by better retrieval. That's fine-tuning territory (or better prompting).

4. Ignoring the eval gap. After fine-tuning, run the SAME eval set on before/after and on a held-out set. Many fine-tunes degrade on tasks outside the training distribution.

5. Fine-tuning when RAG + a better model would work. Sometimes the fix is a stronger base model (Sonnet → Opus) + RAG, not fine-tuning a weaker model.

6. Treating fine-tuning as one-time. Fine-tuned models drift. When your data shifts or a new base model releases, you need to retrain. Budget for it.

7. Fine-tuning removes RAG's need. No — fine-tuning cannot memorize your dynamic knowledge (pricing, policies, catalog). The two are complements, not substitutes.

8. Continuing without evals. Neither RAG nor fine-tuning works without an eval loop. See the LLM Evals journey for the details.

The L4 → L7 architecture ladder

• L4 (starter): Frontier model + prompt engineering + few-shot examples. Simplest possible. Fine for < 10K requests/day.
• L5 (production): RAG with a strong base model. Attribution works, knowledge stays fresh. Standard for most enterprise AI features.
• L6 (advanced): RAG + LoRA fine-tuned smaller base model. Fine-tune for consistent behavior; RAG for knowledge. Cost drops 5-10×.
• L7 (frontier): Full-time model-adaptation pipeline. Continuous data collection → weekly LoRA updates → shadow A/B testing → automated rollout. Fine-tuning is a first-class ML platform, not a one-time project.

Which to pick — the 30-second answer

• Knowledge is dynamic OR large (>100K docs) → RAG.
• Task requires specific behavior/style/decision-making AND you have 1000+ examples AND volume is high → Fine-tuning.
• Both problems exist AND volume justifies it → Hybrid.
• Not sure → RAG first. Always. It's cheaper to get wrong.

What's next in this journey:

• Chapter 1: The data collection playbook — turning historical support tickets, wiki pages, and human feedback into a fine-tuning dataset
• Chapter 2: LoRA fine-tuning end-to-end — using Together AI, Modal, or OpenAI's fine-tune API
• Chapter 3: Evaluating fine-tuned models — the specific eval strategy that catches distribution-drift regressions
• Chapter 4: The hybrid pattern implementation — architecture, deployment, and cost telemetry
• Chapter 5: When to move from LoRA to full fine-tune to distillation
• Chapter 6: Model routing between fine-tuned + frontier — the fallback pattern for edge cases

Sources cited in this chapter:

• Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models" (2021): arxiv.org/abs/2106.09685
• Dettmers et al. "QLoRA: Efficient Fine-tuning of Quantized LLMs" (2023): arxiv.org/abs/2305.14314
• Rafailov et al. "Direct Preference Optimization" (2023): arxiv.org/abs/2305.18290
• OpenAI fine-tuning guide: platform.openai.com/docs/guides/fine-tuning
• Anthropic on when to fine-tune: anthropic.com/news/fine-tune-claude-3-haiku
• Together AI fine-tuning docs: docs.together.ai/docs/fine-tuning-overview
• Hamel Husain — "Fine-tuning is dead": hamel.dev/blog/posts/prompt-engineering
• LlamaIndex fine-tuning docs: docs.llamaindex.ai/en/stable/optimizing/fine-tuning
• OpenAI's model distillation approach: openai.com/research/distillation