Prompt engineering fundamentals

Every engineer I have onboarded starts prompt engineering the same way. They open ChatGPT. They type a request. They read the response. They iterate. They land somewhere OK. Then they copy the "working" prompt into their production code and ship.

Three weeks later they file a bug: "the LLM sometimes returns wrong format." Six months later they file another: "output quality dropped after the model update." A year later they hire a "prompt engineer" to fix everything.

Every one of those bugs is the same class of problem: prompting was treated as art, not engineering.

Prompt engineering is a real engineering discipline. It has patterns, techniques, and trade-offs. Once you learn the four canonical techniques below, you can defend every production prompt with a specific reason: "I chose few-shot with 3 examples because the task shape is unusual and the model needs pattern demonstration." Not "it worked when I tried it."

The 4 techniques at a glance

This is the entire decision tree. Memorize it. Every production LLM prompt is one of these four (or a combination):

Rule: every production prompt maps to ONE primary technique. Combinations are legal (Structured + CoT is common) but you should be able to name the primary.

Now let me walk through each — the visual, the mechanism, the winning scenarios, and the anti-pattern to avoid.

Technique 1: Zero-shot — "just tell the model"

What it is: you describe the task in natural language, no examples. The model uses its training to figure out what you want.

text
    ══════════ ZERO-SHOT PROMPT ANATOMY ══════════

    ┌─────────────────────────────────────────────────┐
    │ SYSTEM: You are a customer support classifier. │  <- persona / role
    │         Classify user messages into one of:    │  <- task definition
    │         [billing, technical, sales, other].    │
    │         Output only the category name.         │  <- output constraint
    └─────────────────────────────────────────────────┘
    ┌─────────────────────────────────────────────────┐
    │ USER: I can't log in and my subscription just  │
    │       renewed. What do I do?                    │
    └─────────────────────────────────────────────────┘
              │
              ▼
    ┌─────────────────────────────────────────────────┐
    │ ASSISTANT: technical                            │  <- clean output
    └─────────────────────────────────────────────────┘

    Cost: ~50 input tokens · 1 output token · $0.0002 per call (GPT-4o)
    Latency: ~200 ms
    Reliability: ~90% on tasks the model already knows well

When zero-shot wins:
- Common tasks the model was heavily trained on: classification, summarization, translation, sentiment analysis
- Tasks you can describe in 1-2 sentences
- You need low latency and low cost
- Reliability >90% is acceptable

When zero-shot loses:
- Custom labels the model has not seen (e.g., internal product tier names)
- Unusual output format (e.g., "output as PostgreSQL VALUES tuple")
- Task where accuracy needs to be >95%

Anti-pattern: wrapping a zero-shot prompt in "You are the world's best expert on X, and you MUST answer perfectly." Adding hype words to the system prompt does not improve accuracy. It just wastes tokens. Reference: Wei et al. (2023) — "Larger language models do in-context learning differently" — larger models increasingly ignore hype and follow the actual task specification.

Interview soundbite: "Zero-shot is my default for standard tasks. Classification, summarization, translation — the model already knows how to do these. I only escalate to few-shot when the task has unusual labels or the model's output format is inconsistent."

Technique 2: Few-shot — "show, don't tell"

What it is: you include 3-5 example (input, output) pairs in the prompt. The model learns the pattern from those examples and applies it to your actual query.

text
    ══════════ FEW-SHOT PROMPT ANATOMY ══════════

    ┌─────────────────────────────────────────────────┐
    │ SYSTEM: Convert user requests to internal      │  <- task frame
    │         ticket categories using our labels.    │
    └─────────────────────────────────────────────────┘
    ┌─────────────────────────────────────────────────┐
    │ USER: I can't reset my password                │
    │ ASSISTANT: tier_1__account_access              │  <- example 1
    │                                                 │
    │ USER: The graph on my dashboard is wrong       │
    │ ASSISTANT: tier_2__data_accuracy               │  <- example 2
    │                                                 │
    │ USER: How do I export my data as CSV?          │
    │ ASSISTANT: tier_1__self_service_help           │  <- example 3
    │                                                 │
    │ USER: My reports are running slower today      │  <- actual query
    └─────────────────────────────────────────────────┘
              │
              ▼
    ┌─────────────────────────────────────────────────┐
    │ ASSISTANT: tier_2__performance                 │  <- pattern followed
    └─────────────────────────────────────────────────┘

    Cost: ~300 input tokens · 5 output tokens · $0.0008 per call (GPT-4o)
    Latency: ~300 ms
    Reliability: ~97% once examples are curated
    Sensitivity: order + diversity of examples matters!

When few-shot wins:
- Custom taxonomies (your internal category names, ticket tiers, product labels)
- Unusual output shapes (a specific delimiter, custom JSON, code in a niche DSL)
- The task has 2-3 subtle edge cases you can demonstrate with examples
- Zero-shot accuracy is <95% and you need higher

When few-shot loses:
- Task is trivial (waste of tokens)
- You cannot curate representative examples (garbage examples in = garbage out)
- Cost is critical (examples inflate input tokens 5-10x)

Anti-pattern: picking examples randomly from your dataset. The examples are load-bearing — biased examples introduce biased outputs. Reference: Zhao et al. (2021) — "Calibrate Before Use: Improving Few-Shot Performance of Language Models" ([arxiv.org/abs/2102.09690](https://arxiv.org/abs/2102.09690)) — the paper showing that example order and label distribution have massive effects on accuracy.

How to pick examples (from the paper above and follow-on work):
1. Cover the label space evenly (one example per class)
2. Cover edge cases you have seen fail zero-shot
3. Order easy → hard (recency bias helps)
4. Keep each example short — don't blow token budget

Interview soundbite: "Few-shot is my next escalation after zero-shot. I curate 3-5 examples covering the label space and the edge cases I have seen fail. Example order matters — I put the hardest last because of recency bias. Reference: Zhao et al. 2021 on example calibration."

Technique 3: Chain-of-Thought (CoT) — "think out loud"

What it is: you instruct the model to show its reasoning BEFORE giving the final answer. The reasoning trace itself becomes context that the model uses to produce a better answer.

text
    ══════════ CHAIN-OF-THOUGHT PROMPT ANATOMY ══════════

    ┌─────────────────────────────────────────────────┐
    │ SYSTEM: You are a shipping cost calculator.    │
    │         Show your work step by step, then      │  <- CoT trigger
    │         output the final cost as a number.     │
    └─────────────────────────────────────────────────┘
    ┌─────────────────────────────────────────────────┐
    │ USER: Ship a 12 kg package from Seattle to     │
    │       Boston, express, insured for $500.       │
    └─────────────────────────────────────────────────┘
              │
              ▼
    ┌─────────────────────────────────────────────────┐
    │ ASSISTANT:                                      │
    │  Step 1: base rate for 12 kg express is $45   │  <- reasoning trace
    │  Step 2: coast-to-coast surcharge is $8       │  (the model uses THIS
    │  Step 3: insurance is 0.5% × $500 = $2.50     │   as context for its
    │  Step 4: total = 45 + 8 + 2.50 = $55.50       │   final answer)
    │                                                 │
    │  Final cost: $55.50                            │  <- constrained output
    └─────────────────────────────────────────────────┘

    Cost: ~350 input tokens · ~80 output tokens · $0.001 per call (GPT-4o)
    Latency: ~800 ms (reasoning trace = more tokens to generate)
    Reliability: 10-30 percentage points BETTER than zero-shot on
                 multi-step math/logic/planning tasks
    Trade-off: 4x more output tokens = 4x higher LLM cost

When CoT wins:
- Multi-step math or logic (Wei et al. 2022 showed +30 points on GSM8K)
- Planning tasks (agent action selection)
- Compliance / eligibility checks (rule-based reasoning)
- Anywhere the interviewer's question is "does the model reason?" — CoT provides the trace

When CoT loses:
- Simple lookups (extra tokens waste $ and add latency)
- Latency-critical UX (users see the reasoning slowly stream in — annoying for simple tasks)
- The model's reasoning is confidently wrong (CoT can also confidently rationalize a wrong answer — see "hallucination")

The most cited paper in LLM history: Wei et al. (2022) — "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" ([arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903)). The paper that showed adding "Let's think step by step" improves math accuracy from 18% → 57% on GSM8K. This 4-word phrase was one of the most impactful discoveries in prompting.

Modern variant — internal reasoning tokens: OpenAI's o1 and o3 series, Anthropic's Claude 3.5 Sonnet with extended thinking, and Google's Gemini 2.5 have built-in CoT that runs BEFORE the visible output. You pay for reasoning tokens even if you don't see them. Reference: [openai.com/index/introducing-openai-o1-preview](https://openai.com/index/introducing-openai-o1-preview) (Sep 2024).

Interview soundbite: "For multi-step reasoning I use chain-of-thought — 'show your work step by step' before the final answer. Wei et al. 2022 showed a 30-point accuracy jump on math tasks. I pay for it in output tokens and latency. Modern reasoning models (o1, Claude with extended thinking, Gemini 2.5) do this internally so I don't have to prompt it — but I still pay for the reasoning tokens."

Technique 4: Structured Output — "give me exactly this shape"

What it is: you constrain the model to produce output matching a specific schema (JSON, XML, function-call arguments). The provider's runtime forces the output to match — not just asks.

text
    ══════════ STRUCTURED OUTPUT ANATOMY ══════════

    ┌─────────────────────────────────────────────────┐
    │ SYSTEM: Extract entities from the message.     │
    │                                                 │
    │ SCHEMA (enforced at runtime):                  │  <- passed as separate
    │  {                                              │     parameter to API,
    │    "customer_name": string,                    │     NOT prompt text
    │    "product": string,                          │
    │    "sentiment": "positive"|"negative"|"neutral"│
    │  }                                              │
    └─────────────────────────────────────────────────┘
    ┌─────────────────────────────────────────────────┐
    │ USER: Hi, I'm Sarah Chen and I love the new    │
    │       dashboard update!                         │
    └─────────────────────────────────────────────────┘
              │
              ▼  (LLM output is constrained by schema at token-decode time)
    ┌─────────────────────────────────────────────────┐
    │ ASSISTANT:                                      │
    │  {                                              │
    │    "customer_name": "Sarah Chen",              │
    │    "product": "dashboard update",              │
    │    "sentiment": "positive"                     │
    │  }                                              │
    └─────────────────────────────────────────────────┘

    Guaranteed: output IS valid JSON matching the schema.
    Guaranteed NOT: the values are semantically correct - that's
    still the model's job. Structure enforcement != correctness.

When structured output wins:
- Every time you parse the output programmatically (JSON.parse, XML deserialize)
- Anywhere a downstream tool expects a specific shape (function-calling)
- Agent action selection (the "tool call" schema in ReAct — see A4)
- Data extraction pipelines

When structured output loses:
- Free-form generation (essay, chat response, translation) — schema is a straitjacket
- The schema is too complex (deeply nested unions can confuse the model)

The three ways to get structured output:

1. Native JSON mode (OpenAI response_format: { type: "json_object" }, Anthropic Claude with tools, Gemini structured output). Guaranteed valid JSON but not schema-compliant.

1. Constrained schemas (OpenAI response_format: { type: "json_schema", ... } released Aug 2024; Anthropic tools). Guaranteed valid JSON AND matching a specific schema. Reference: [platform.openai.com/docs/guides/structured-outputs](https://platform.openai.com/docs/guides/structured-outputs).

1. Function calling / tool use (OpenAI functions, Anthropic tools, Gemini function calling). Schema is a tool signature; output is a function invocation. Best for agent workflows.

Anti-pattern to burn into your brain: writing "output as JSON" in the system prompt and hoping. The model will 95% of the time produce valid JSON. That 5% ships to production. Users see errors. On-call pages you at 3AM. Use the API-level structured output flag. Every. Time.

Interview soundbite: "For any output my code parses, I use structured output — OpenAI json_schema mode, Anthropic tools, or function calling. Never 'ask for JSON in the prompt.' The API-level flag guarantees valid schema-matching output at token-decode time; the prompt-level ask is a 5%-failure-rate coin flip that will page you at 3AM."

The prompting workflow evolution (L4 → L7)

Prompting is not just about the ONE prompt — it's about how prompts are AUTHORED, REVIEWED, TESTED, DEPLOYED, MONITORED. Scale drives the workflow, not just the technique:

Read this chart carefully. It tells you two things a newbie must internalize:

1. The technique (zero/few/CoT/structured) matters LESS than the workflow around it. A great prompt with no eval is a ticking bomb. A mediocre prompt with rigorous eval + rollback is safe.
2. The workflow must match the scale. Building L6 tooling for an L4 project is over-engineering. Shipping L4 casualness at L6 traffic is negligence.

The Chapter roadmap

Chapters 1-4 — foundations:
- Ch 1 Requirements — the 6 questions before you write any prompt
- Ch 2 Prompt anatomy — system / user / assistant roles, delimiters, context window budget
- Ch 3 System prompts — the seat of persona, constraints, output rules
- Ch 4 The 4 techniques deep-dive — zero-shot, few-shot, CoT, structured — with production examples

Chapters 5-8 — the workflow evolution shown above.

Chapters 9-12 — failures, evaluation, safety, production:
- Ch 9 When prompts fail — hallucination, refusal, format drift, prompt injection
- Ch 10 Prompt evaluation — golden sets, LLM-as-judge, human labeling
- Ch 11 Prompt versioning, A/B testing, canary rollout
- Ch 12 Defense — interview questions on prompting

Chapters queued (session note)

Chapters 1-12 of this journey are queued for follow-on authoring sessions. This Chapter 0 exists so you can see the 4-technique decision tree and internalize the workflow-scale mapping. Come back as remaining chapters ship.

Newbie mentor commentary — read this every time you write a prompt

1. Every prompt is engineering, not art. Name the technique you are using (zero / few / CoT / structured). Justify why. Have a fallback plan when it fails.

1. Structured output is not optional above trivial prototypes. The 5% failure rate of "ask for JSON in the prompt" is what pages you at 3AM. Use the API-level flag. Every provider has it now.

1. Chain-of-thought costs money. 4-10× more output tokens for the reasoning trace. Only pay for it when the accuracy gain (or the required reasoning transparency) justifies the cost.

1. Few-shot examples are load-bearing code. Curate them like unit tests. Cover the label space. Cover the edge cases. Never pick randomly.

1. The prompt is the smallest part of your prompt engineering. The eval harness, the version control, the A/B rollout mechanism — those are 90% of the work. Ch 10 and 11 will make that concrete.

References for what's coming

• Wei et al. (2022) — "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (arxiv.org/abs/2201.11903) — the CoT paper.
• Brown et al. (2020) — "Language Models are Few-Shot Learners" (arxiv.org/abs/2005.14165) — the GPT-3 paper that established few-shot as a primary technique.
• Zhao et al. (2021) — "Calibrate Before Use: Improving Few-Shot Performance of Language Models" (arxiv.org/abs/2102.09690) — example order matters.
• OpenAI Structured Outputs (Aug 2024) — openai.com/index/introducing-structured-outputs-in-the-api — the guaranteed-schema flag.
• Anthropic Prompt Engineering Guide — docs.anthropic.com/en/docs/build-with-claude/prompt-engineering — vendor-official best practices.
• OpenAI Prompt Engineering Guide — platform.openai.com/docs/guides/prompt-engineering — vendor-official best practices.
• Google Vertex AI Prompt Engineering — cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/introduction-prompt-design — Gemini-specific patterns.
• Braintrust, LangSmith, Humanloop — the three commercial prompt-management platforms covered in Ch 6+.
• DSPy — dspy.ai — Stanford's framework for auto-optimized prompts (L7).
• Chapter 3 of A3 RAG journey — RAG prompts are just structured-output prompts with retrieval context; the same techniques apply.
• Chapter 3 of A4 Agent journey — agent tool schemas are just structured-output specifications; same discipline.