Skip to main content
load-balancer

Load Balancer

Distributes incoming traffic across a pool of servers for scale and fault tolerance.

Why it exists

One server can only do so much. When traffic exceeds the capacity of a single machine, or when you need redundancy so a single crash doesn't take you down, you put a load balancer in front and add servers behind it. The LB becomes the single entry point and turns a fleet into what looks like one big machine.

How it works

Layer 4 (TCP/UDP) load balancers route packets by connection tuple; they're fast and lightweight but see none of the HTTP semantics. Layer 7 (HTTP) balancers terminate the connection, inspect headers/paths/cookies, and route intelligently — sticky sessions, header-based routing, canary splits, TLS termination. Modern LBs also do health checks (removing dead nodes automatically), connection draining (letting existing requests finish before a node is retired), and rate limiting.

Scaling characteristics

A single LB is horizontally scalable via DNS round-robin, Anycast IP, or a chained LB (L4 in front of L7). Cloud LBs scale automatically; self-managed (Nginx, HAProxy, Envoy) scale via a shared VIP or Anycast. LB throughput is measured in packets per second (L4) or requests per second (L7); a modern Envoy on commodity hardware handles ~50-100K RPS per instance.

When to use it

  • You have more than one instance of any service (which is almost always)
  • You need zero-downtime deploys via canary or blue-green
  • You want to move TLS termination out of your application
  • You need path- or header-based routing to different upstreams

When NOT to use it

  • You truly have a single instance (dev env only) — direct DNS is fine
  • Latency budget is sub-millisecond and even one hop is too much (rare — LBs add ~1ms)

Failure modes

  • LB itself becomes a single point of failure — mitigate with 2+ LBs behind Anycast or a shared VIP
  • Health check misconfiguration marks healthy nodes as dead (or vice versa), cascading to full outage
  • Sticky sessions pin traffic to failed nodes; when the node dies, sessions are lost
  • Connection storm on LB restart — clients reconnect in a synchronized wave, saturating the new LB

Alternatives

  • DNS-based load balancing (round-robin A records) — simplest, but no health checks, TTL-limited failover
  • Client-side load balancing (gRPC does this) — no middleman hop, but clients need discovery + retry logic
  • Service mesh sidecars (Envoy/Linkerd per pod) — great for east-west traffic; still often need an ingress LB

Interview questions

  • L4 vs L7 — when would you use each?
  • How does your LB handle a viral spike that saturates connections?
  • What does a health check look like — active vs passive, and what interval?
  • What happens when the LB itself goes down?
  • How do you drain connections before retiring a node?
  • What breaks first at 100K RPS through a single LB?

Systems that use this component

See how the real designs on this platform put load-balancer to work — concrete usage context per system.

URL Shortener

beginner

AWS ALB terminates TLS and routes to 3 stateless app servers

Open system

Twitter/X timeline

intermediate

L7 ALB distributes API traffic across microservices

Open system

Netflix

intermediate

Distributes control-plane traffic to service mesh; data-plane goes via Open Connect

Open system

WhatsApp

intermediate

Terminates client TLS then routes to chat servers based on user hash

Open system

Payment System

intermediate

Sticky sessions for payment flows during checkout

Open system

YouTube

intermediate

Routes upload/api traffic to origin; video traffic to CDN

Open system

Instagram

intermediate

L7 gateway with rate limiting per user

Open system

Slack

intermediate

Distributes WebSocket connections to chat servers

Open system

Dropbox

intermediate

Splits metadata API traffic vs block storage traffic

Open system

Ride Sharing

intermediate

Regional ALB per city, with cross-region failover

Open system

Ticket Booking

intermediate

Handles seat-reservation traffic bursts during on-sale events

Open system
DEEP DIVE — INTERACTIVE
Interactive walkthrough with animated visuals

In 1996, Cisco engineers built the first commercial hardware load balancer: LocalDirector. Before it, if your website got popular, you had one option — buy a bigger server (vertical scaling). LocalDirector let you put 10 cheap servers behind a device that spread traffic across them, and if one failed, the device stopped sending traffic to it. Suddenly you could scale out, not just up.

Today load balancers are foundational infrastructure. AWS ALB, Cloudflare, Nginx, HAProxy, Envoy, Google Cloud LB, F5 BIG-IP — all descendants of that 1996 idea. They live at Layer 4 (TCP/UDP — fast, protocol-agnostic) or Layer 7 (HTTP-aware — path/header/cookie routing, TLS termination). Every serious system-design interview will ask which and why.

Historical framing

  • 1996 — Cisco LocalDirector, first commercial hardware LB.
  • 1998 — F5 BIG-IP, enterprise LB that defined the L7 category.
  • 2003 — HAProxy by Willy Tarreau, OSS L4/L7 that still runs half the internet.
  • 2004 — Nginx by Igor Sysoev, OSS reverse proxy + L7 LB. Now runs ~35% of the web.
  • 2016 — Envoy from Lyft, modern C++ L7 proxy. Foundation of Istio/Linkerd/Consul Connect.

Interactive: L4 vs L7 side by side

The most important decision. Watch the same HTTPS request traverse each type and see what each layer can (and can't) do.

Same request, two layers — step 1 of 9
Watch how L4 and L7 handle the same HTTPS request differently.
L4 · Network Load Balancer
1. TCP conn
2. Route by IP:port
3. Forward packets
4. Backend
Fast, protocol-agnostic. Cannot inspect content.
L7 · Application Load Balancer
1. TCP conn
2. Terminate TLS
3. Parse HTTP
4. Route by /path or Host
5. New TCP to backend
6. Backend
Rich routing, TLS termination. ~5-10× slower than L4.
L4: client opens TCP connection to LB IP.

Interactive: distribution algorithms

Pick a strategy, watch 12 incoming requests distribute across 4 backends. Each algorithm has failure modes.

Assign each request to the next backend in order. Simple, evenly distributes if all backends are equal.
B1
3
of 12 requests
B2
3
of 12 requests
B3
3
of 12 requests
B4
3
of 12 requests

Health checks — how bad nodes get removed

A load balancer without health checks is worse than no load balancer — because dead nodes silently drop requests. Modern LBs support active (LB periodically probes) and passive (LB watches actual responses for errors) health checks.

Health check state machine
Healthy
Passing checks. LB sends traffic. Threshold: 2 consecutive successes to enter this state.
Draining
Being taken out gracefully. LB stops sending new connections but lets in-flight requests finish.
Unhealthy
Failed 3 consecutive checks. LB stops sending traffic. Continues probing to catch recovery.
Active check: LB actively probes each node every N seconds (e.g., HTTP GET /health).Passive check: LB watches real request outcomes — after M consecutive 5xx responses, mark unhealthy.

Typical settings: 3 consecutive failures within 30 seconds = node marked unhealthy. 2 consecutive successes = healthy again. Too aggressive and you flap on transient blips; too lenient and dead nodes serve errors for minutes.

Connection draining — retiring a node gracefully

You need to deploy new code to server X. You cannot yank X immediately — 500 in-flight requests would fail. Load balancers support connection draining: stop sending new connections to X, let existing ones finish (timeout typically 30-300 seconds), then remove X.

Connection draining — step 1 of 5
Retiring node X gracefully.
Node W
✓ new
Node X (draining)
✓ new
12 in-flight
Node Y
✓ new
Steady state — node X serves ~12 in-flight requests plus new arrivals.

L7 routing rules — the flexibility

L7 load balancers can route based on almost anything in an HTTP request. Common patterns:

Path prefix
path /api/*
api-service
GET /api/users → api-service
Host header
Host: admin.example.com
admin-backend
Different subdomains → different services
Header value
X-Canary: true
canary-deployment
Canary rollout via header flag
Query string
?feature=beta
beta-backend
Feature-flag testing
Cookie
user-tier=premium
premium-backend
Sticky routing to a premium tier
Method
POST
write-backend
CQRS: reads/writes to different backends

Product comparison — pick the right LB

ProductLayerTLS terminationTypical latencyCostBest for
AWS ALBL7Yes (ACM)~1-3ms$Most AWS HTTP workloads
AWS NLBL4Passes through~100µs$$gRPC, Postgres, Redis, non-HTTP
CloudflareL7Yes (universal)~5ms edgeFree tier okGlobal apps + DDoS protection
NginxL4/L7Yes (config)~1msOSS freeSelf-hosted, static-heavy sites
HAProxyL4/L7Yes~1msOSS freeHigh-QPS OSS deployment
EnvoyL7Yes (mTLS strong)~1-2msOSS freeService mesh, gRPC, sidecar
F5 BIG-IPL4/L7Yes~1ms$$$Enterprise, iRules power users
Google Cloud LBL7Yes~1-3ms$GCP-hosted apps

Applied in real systems

AWS ALB

AWS Application Load Balancer — the AWS default L7

L7. Routes by path, host, method, header, query. Terminates TLS via ACM. Integrates with WAF, Cognito, Lambda. Charged per request + LCU. Standard choice for most AWS-hosted HTTP workloads.

AWS NLB

AWS Network Load Balancer — pure L4

L4 TCP/UDP. Preserves source IP. Handles millions of connections/sec at ~100µs latency. Cheaper than ALB per LCU. Use for gRPC, Postgres, Redis, anything non-HTTP.

Cloudflare

Cloudflare — the global Anycast LB

300+ data centers. Anycast IP means every user hits the nearest PoP. Same TLS termination, WAF, caching, rate limiting as ALB, but at edge. Free tier absorbs 20% of internet traffic.

Nginx

Nginx — 30% of internet web servers

L7 reverse proxy + LB. Config via nginx.conf. Handles static assets, TLS termination, upstream health, cache. Used by Netflix, Airbnb, GitHub, Dropbox as their edge tier.

HAProxy

HAProxy — the OSS reference

L4 or L7. Blazingly fast. Willy Tarreau's code has run half the internet since 2003. Reddit, StackOverflow, GitHub, Twitter (historically) — all HAProxy. Config is dense but powerful.

Envoy

Envoy — the service mesh sidecar

L7 proxy from Lyft (2016). Runs as a sidecar per pod in Istio, Linkerd, Consul Connect. Handles mTLS, retries, circuit breakers, observability. Kubernetes service mesh standard.

F5 BIG-IP

F5 BIG-IP — enterprise hardware/virtual LB

Enterprise's choice. Rich policy engine (iRules). Common in banks, healthcare, government. Hardware appliances cost $50K+; virtual editions available.

Google Maglev

Google Maglev — L4 at planet scale

Google's internal L4 LB (NSDI 2016 paper). Runs Google.com, YouTube, Maps. Software-based, consistent hashing to backends. Millions of packets per second per instance.

Key takeaways

  • L4 vs L7 is the single biggest LB decision. L4 = fast + protocol-agnostic (NLB). L7 = HTTP-aware + slower + more expensive (ALB, Nginx, Envoy).
  • Distribution algorithms: round-robin (default), least-connections (uneven request sizes), weighted, consistent hash (cache affinity), IP hash (session stickiness).
  • Health checks are non-negotiable. Active = proactive; passive = reactive. Tune to avoid flapping on transient blips.
  • Connection draining lets you retire nodes without dropping in-flight requests. 30-300s timeout typical.
  • TLS termination at the LB is the standard pattern. Origin sees plain HTTP; LB handles cert rotation + TLS 1.3 upgrades.
  • Every serious architecture has at least 2 LBs — often L4 in front (Anycast/NLB) with L7 behind (ALB/Nginx). Never a single point of failure.

References

  • Eisenbud et al. (2016) — Maglev: A Fast and Reliable Software Network Load Balancer. NSDI.
  • Envoy design docs — envoyproxy.io
  • HAProxy documentation — the definitive practical reference at haproxy.com/documentation.
  • Nginx official docs — nginx.org/en/docs/.
  • AWS ALB/NLB whitepapers — comparison + best practices at docs.aws.amazon.com/elasticloadbalancing.