Latency vs throughput
Two axes that pull in opposite directions.
Latency is the time to serve one request. Throughput is how many requests you serve per second. They're not the same. And here's the cruel part: they trade off against each other. Increase concurrency to boost throughput, and per-request latency shoots up. Reduce concurrency for lower latency, and throughput drops.
Every system-design decision touches this trade-off. Netflix and YouTube optimize for throughput (huge volumes of video bytes). High-frequency trading and gaming optimize for latency (microseconds matter, throughput per box irrelevant). Most transactional web apps sit in the middle — hit both targets, live in the messy middle.
The math — Little's Law
Little (1961) — L = λ × W. In a stable system, the average number of items in the queue (L) equals the arrival rate (λ) times the average time in the system (W). Simple, elegant, universal. It means: if you know throughput and latency, you know queue depth. If two of the three are set, the third is fixed.
Interactive: watch latency explode as concurrency rises
Move the concurrency slider. Watch what happens to latency. (This is a simulation of the M/M/1 queueing model — real systems follow similar curves.)
The knee: at ~80% utilization, latency starts climbing steeply. At 95% utilization, it's catastrophic. This is why production systems target 60-70% CPU, not 95%. The remaining headroom is not waste — it's the buffer against traffic spikes and GC pauses.
Percentiles matter more than averages
"Average latency 20ms" hides a lot. What's the P99? The P999? If your average is 20ms but P99 is 500ms, 1% of users have a bad time. At 100k RPS, that's 1000 unhappy users per second.
The tail latency problem — why the slowest matters
Dean & Barroso's 2013 paper "The Tail at Scale" showed that when you fan out one request to N backends, your response is bottlenecked by the slowest backend. At N=100 fan-out and each backend's P99 = 100ms, your P99 aggregate is near 1s — nearly always some backend is having a bad day.
Mitigations: hedged requests (send to two backends, use the first response), backup requests (send delayed second request if first is slow), and capacity headroom (never load past 70%).
The three latency numbers you must memorize
| Operation | Time | Human scale |
|---|---|---|
| Register / L1 cache | 1 ns | human blinks |
| L2 cache | 3 ns | |
| RAM access | 100 ns | |
| SSD read | 100 μs | reads a paragraph |
| Round trip within DC | 500 μs | |
| SSD write + fsync | 1 ms | |
| HDD seek | 10 ms | |
| Round trip US East → US West | 70 ms | |
| Round trip US → EU | 80 ms | |
| Round trip US → Asia | 150 ms | |
| Fast response perceived as instant | <100 ms | human threshold |
| Fast page load | <1 s |
Applied in real systems
High-frequency trading — nanosecond latency
Bloomberg, Jane Street, Jump Trading. Latency budgets under 1 microsecond. Colocated servers next to exchange racks. FPGAs. Kernel bypass (DPDK). Every nanosecond = money.
Fortnite, League of Legends
UDP + game engines optimized for 30-50ms latency. 60 FPS = 16.6ms per frame budget. Regional matchmaking to keep RTT low.
YouTube — throughput-first
Not latency-sensitive after initial buffer. Streams optimized for aggregate GB/s to millions of viewers. Latency of first byte matters (~200ms goal); once buffered, throughput is king.
Netflix — Open Connect + prebuffer
Same as YouTube — throughput dominates. Serves video from ISP-embedded caches. Aggregate egress: petabytes/day. Client prebuffers 10-30s so per-chunk latency is invisible.
Google Search — sub-200ms with fan-out
Every search fans out to hundreds of index shards. Uses hedged requests to keep P99 below 200ms even when individual shards P99 is 100ms+.
CDNs — latency at edge
Cloudflare, Fastly, Akamai. Every optimization is about reducing user-perceived latency. Edge caches, Anycast, HTTP/3, TLS 1.3 all shave milliseconds.
Key takeaways
- Latency = time for one request. Throughput = requests per second. Independent axes.
- They trade off. Higher concurrency = higher throughput but higher per-request latency.
- Little's Law: L = λ × W. Two of three set the third.
- Utilization knee at ~80%. Run production at 60-70%. Remaining is headroom, not waste.
- Percentiles matter. Report P50, P95, P99. Averages hide tail problems.
- Tail at scale: fan-out amplifies tail latency. Hedge to mitigate.
- Different systems optimize different dimensions. HFT: latency. YouTube: throughput. Web: both.
References
- Little (1961) — Little's Law.
- Dean & Barroso (2013) — "The Tail at Scale." CACM.
- Kingman (1970s) — Kingman's formula for M/G/1 queue.
- Peter Bailis and other queueing-theory writers online.
Practice what you just read
Every foundation concept has a companion quiz to close the loop.