LLM evals that matter

You built an LLM feature. It works "well" in demos. Now you need to answer a real engineering question:

"Is version 2 of my prompt better than version 1?"

You can't A/B test on real users yet (chicken-and-egg: you need confidence to ship). You can't just read 10 outputs — sampling is unreliable and you'll flip a coin on close calls. You need a metric.

You look up "LLM evaluation." Papers say BLEU. ROUGE. METEOR. BERTScore. You compute them. Version 2 scores 0.71, Version 1 scores 0.68. Ship? You look at the outputs. Version 1 answers questions better. Version 2 hallucinates confidently. The metric said the opposite.

This is the eval crisis every LLM engineer runs into. Classical NLP metrics were designed for machine translation and summarization — tasks where "closer to a reference string" meant "better." Modern LLM tasks (Q&A, agents, tool use, chat) have no single reference. There are many good answers. There are subtly bad answers that string-match well.

The industry moved past BLEU/ROUGE two years ago. What replaced them is not one metric — it's a layered evaluation strategy, where each layer catches a different class of failure. Once you understand the pyramid, you can defend every ship decision with the right layer of evidence.

The 4-layer eval pyramid — the only diagram you need

Rule: every change ships through the layers bottom-to-top — deterministic assertions block bad changes fast and cheap; LLM-as-judge catches quality regressions; humans calibrate the judge; live signal catches everything else. Skip any layer and the layer above starts catching what should have been caught below (expensive) — or the failure escapes to users (very expensive).

Let me walk through each layer.

Layer 1: Deterministic assertions — pytest for LLMs

What it is: you write assertions that a valid output must satisfy. No LLM involved. No judgment call. Pass or fail.

Anatomy:

python
    def test_json_extraction(model_response: str):
        # 1. Parses as JSON
        data = json.loads(model_response)  # raises → fail

        # 2. Has required fields
        assert "customer_email" in data
        assert "priority" in data

        # 3. Field values are in valid enum
        assert data["priority"] in ["low", "medium", "high"]

        # 4. Email looks like an email
        assert re.match(r"^[^@]+@[^@]+\.[^@]+$", data["customer_email"])

        # 5. Response is under 500 tokens
        assert len(response.split()) < 500

What catches: JSON that doesn't parse, missing fields, enum mismatches, empty responses, refusals ("I can't help with that"), oversized outputs, calls to disallowed tools, PII leakage of specific patterns.

Cost: microseconds. No API calls. Run on every commit.

Limitation: cannot judge quality. An output that's technically correct but unhelpful will pass Layer 1.

Coverage estimate: Layer 1 typically catches 20-40% of real production failures. Cheap and worth it — but never sufficient alone.

Layer 2: LLM-as-judge — a stronger model grades your model

What it is: you use a strong LLM (usually GPT-4o, Claude 3.5+ Sonnet, or Gemini Pro) to grade outputs from your (usually smaller/cheaper) production model. The judge receives the question, the reference (if you have one), the model's answer, and a rubric — and returns a score plus reasoning.

Anatomy:

python
    JUDGE_PROMPT = """
    You are an evaluator. Grade the assistant's answer against the reference.

    Question: {question}
    Reference (an example of a good answer): {reference}
    Assistant's answer: {answer}

    Score on a 1-5 scale:
      1 = wrong or harmful
      2 = incomplete or misleading
      3 = correct but unhelpful
      4 = correct and helpful
      5 = correct, helpful, and better than the reference

    Return JSON: {"score": int, "reasoning": string}
    """

    response = judge_model.chat(JUDGE_PROMPT.format(...))
    result = json.loads(response)
    assert result["score"] >= 4  # threshold you set

What catches: semantic correctness, tone, helpfulness, harmful content, faithfulness (does the answer stay grounded in the retrieved context?), instruction following.

Cost: ~$0.001 per eval on GPT-4o with well-designed prompts. At 500 evals per commit that's $0.50 — cheap.

Limitations that trip everyone up:

• Positional bias — judge prefers the first answer if you compare two side-by-side. Fix: randomize order.
• Length bias — judge prefers longer answers even when shorter is better. Fix: normalize length in the rubric or use pairwise comparison with a "which is more concise?" prompt variant.
• Self-preference bias — judge model rates outputs from the same model family more favorably. Fix: use a different model family as judge (Claude judging GPT outputs, or vice versa).
• Rubric drift — small prompt changes to the judge change scores meaningfully. Fix: version the judge prompt like production code.

Coverage estimate: Layer 2 catches 60-80% of real failures when combined with Layer 1. Not perfect — the judge has real error rates on ambiguous cases — which is why we need Layer 3.

Layer 3: Human review — ground truth for calibrating the judge

What it is: a domain expert (or a curated pool) reviews a sampled subset of outputs and provides ground-truth labels. You then compare the judge's scores against human scores. If the judge disagrees with humans >15% of the time, the judge is unreliable — fix its rubric or use a different judge.

Anatomy:

text
    ══════════ HUMAN-CALIBRATED LLM-AS-JUDGE ══════════

    Step 1: Human labels 100 outputs from your prod system.
            Score 1-5 with brief rationale.

    Step 2: LLM judge scores the same 100 outputs
            using the rubric you've written.

    Step 3: Compute agreement:
              - Exact match rate: 62% (should be >70%)
              - Cohen's kappa: 0.51 (should be >0.6 for reliability)
              - Off-by-1 rate: 89% (usually acceptable if kappa is good)

    Step 4: Investigate DISAGREEMENTS.
            Are they on ambiguous cases or systematic?
            Systematic → improve rubric.
            Ambiguous → accept the noise floor.

    Step 5: Once judge agreement is > 70% with humans,
            use judge in CI at 100x the volume.
            Re-calibrate quarterly.

What catches: rubric ambiguity, edge cases the judge got wrong, drift as the domain evolves.

Cost: ~$10-50 per human-labeled example if you pay a domain expert. ~$1-5 if you use platforms like Prolific or Scale AI for less-specialized tasks.

Limitation: slow. You can't get 10,000 human-labeled examples in a week. So you spot-check the judge with ~100 examples per release cycle.

The insight: humans are the ground truth. LLM-as-judge is a fast, cheap proxy. Layer 3 tells you HOW GOOD your proxy is. Without Layer 3, your Layer 2 scores could be systematically wrong and you'd never know.

Layer 4: Live production feedback — the long-tail catcher

What it is: you instrument production. Thumbs up/down on every output. Time-to-abandonment. Re-ask rate. Conversion. NPS. Session length. Whatever your product's north-star metric is.

Anatomy:

text
    ══════════ LIVE PRODUCTION FEEDBACK LOOP ══════════

    Every LLM output → logged with:
      - Input (redacted for PII)
      - Output
      - Model version + prompt version
      - User feedback (thumbs, retry, edit, dismiss)
      - Downstream signal (did user act on the answer?)
      - Latency + cost
                           ↓
    Aggregated to per-slice metrics:
      - Per user cohort (new/pro/enterprise)
      - Per language / locale
      - Per query intent (extracted via clustering)
      - Per model version
      - Per prompt version
                           ↓
    Alerts on regressions:
      - Thumbs-down rate up >5% for cohort X
      - Abandon rate up >2% overall
      - A specific model version underperforming
                           ↓
    Feedback loop:
      - Bad outputs → into training/eval set
      - New failure modes → new Layer 1 assertions
      - New rubric criteria → into judge prompt

What catches: drift, long-tail failures your eval set didn't cover, changes in user behavior, model degradation from provider updates.

Cost: low ongoing, high initial (need real instrumentation).

Limitation: slow signal — user complaints take days to accumulate. And thumbs-down rates are famously noisy. Combine with quantitative signals (retry rate, session length) for reliability.

The eval pyramid in action — how a change ships

Every commit runs Layers 1+2 (fast). Layer 3 runs async on release candidates. Layer 4 runs continuously post-ship.

The golden-dataset loop — how your eval set gets good

Your eval set is more valuable than your model. It's what lets you make progress. Here's how it grows:

1. Seed — 50-100 hand-crafted examples covering happy path + known edge cases.
2. Failure harvest — every Layer 4 thumbs-down goes into the eval set with a human label.
3. Adversarial add — every prompt-injection attempt, jailbreak, or CVE-caliber input goes in.
4. Coverage sampling — cluster real user queries; sample from underrepresented intent buckets.
5. Version the set — treat it like code; PR reviews for additions; track what's covered.
6. Rotate — retire examples that all models pass; the set drifts toward "hard cases."

After 6 months of this discipline, a well-tended eval set is worth more than half your model choice.

Which metrics still work? (spoiler: not many)

• BLEU/ROUGE/METEOR — useful for translation and summarization vs a reference. Nearly useless for Q&A, agents, chat.
• BERTScore/BLEURT — semantic similarity to reference. Better than BLEU. Still requires a good reference. Useless for open-ended tasks.
• G-Eval / LLM-as-judge — the modern default for open-ended tasks. Requires calibration.
• RAGAS metrics — for RAG specifically: faithfulness, answer relevance, context precision, context recall. Good subset for retrieval systems.
• DeepEval / promptfoo / OpenAI Evals / Langsmith / Arize Phoenix — frameworks for running the above at scale. Pick one and stick with it.

The L4 → L7 eval maturity ladder

• L4 (starter): a few smoke-test assertions + eyeballing sampled outputs. Fine for prototypes.
• L5 (production): all 4 layers, 500-1000 eval cases, LLM-as-judge in CI, human calibration once. Standard for any real product.
• L6 (advanced): multi-model judge ensembles, pairwise comparison + Bradley-Terry ranking, per-slice metrics with alerting, adversarial eval set with weekly additions.
• L7 (frontier): offline + online + shadow evals, real-time canary analysis, budgeted eval spend per commit, automated failure clustering, RLHF loops feeding failures back into training data.

The most common mistakes

1. "We ship on BLEU/ROUGE." These fail for modern LLM tasks. Every 3 months I see a team discover this the hard way. Move to Layer 2 immediately.

2. "One eval score summarizes quality." No. A single number hides that v2 got much better at Q&A but slightly worse at refusals. Report per-slice metrics.

3. "Our judge model is always right." No. Layer 3 exists to catch systematic judge errors. Skip Layer 3 and your entire eval story is built on sand.

4. "We'll add evals after we ship." By that point, you can't tell if this week's regression is real or noise. Set up Layers 1-2 before the first ship.

5. "The eval set can be small — we'll catch things in prod." 20 eval cases doesn't cover long-tail behavior. Aim for 500+ before shipping anything non-trivial.

What's next in this journey:

• Chapter 1: Writing the judge prompt — with a real rubric, positional-bias fixes, and multi-judge ensembling
• Chapter 2: RAGAS in depth — faithfulness, answer relevance, context precision, and context recall with code
• Chapter 3: Pairwise comparison + Bradley-Terry — the ranking approach that beats absolute scores
• Chapter 4: The golden dataset playbook — clustering user queries, coverage analysis, versioning
• Chapter 5: DeepEval, promptfoo, OpenAI Evals — hands-on with the framework you'll actually use
• Chapter 6: Adversarial evals — building a prompt-injection test suite, jailbreak evals, refusal quality

Sources cited in this chapter:

• Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023): arxiv.org/abs/2306.05685
• Liu et al. "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" (2023): arxiv.org/abs/2303.16634
• RAGAS framework: docs.ragas.io
• OpenAI Evals: github.com/openai/evals
• Anthropic evaluation cookbook: docs.anthropic.com/claude/docs/build-tests-and-evals
• Chip Huyen — "Evaluating LLM applications": huyenchip.com/2024/07/25/genai-platform.html
• Hamel Husain — "Your AI product needs evals": hamel.dev/blog/posts/evals
• DeepEval: docs.confident-ai.com
• promptfoo: promptfoo.dev
• LangSmith: docs.smith.langchain.com
• Arize Phoenix: docs.arize.com/phoenix