Rate Limiter

Why it exists

Every service has a capacity ceiling. Without a rate limiter, one runaway client — buggy retry loop, aggressive scraper, DDoS — takes the whole service down. Rate limiting enforces fairness (no one caller consumes disproportionate capacity) and protection (the system stays healthy for the many even when one caller misbehaves). It's a required layer in any public API and many internal ones.

How it works

The rate limiter maintains a counter per caller key (API key, IP, user ID). On each request, it increments the counter and checks against the budget. If over, it returns 429 (Too Many Requests). Algorithms vary: fixed window (simplest, has burst issues at window boundaries), sliding window (smoother, more accurate), token bucket (allows bursts up to bucket size, refills at a rate), leaky bucket (smooths bursts to a constant output rate). Distributed rate limiters use Redis with INCR + EXPIRE.

Scaling characteristics

Local (per-instance) rate limiters are cheap but inaccurate at scale — 10 instances each allowing 100 RPS = 1000 RPS actual. Distributed rate limiters (Redis-backed) are accurate but add 1-2ms per request. Modern designs approximate globally with periodic sync: each instance has a local quota, syncing with a central store every N seconds — accurate to within a few percent, near-zero latency.

When to use it

• Any public API (per-key quotas by tier)
• Login endpoints (per-IP rate limits to prevent brute force)
• Expensive endpoints (search, ML inference)
• Anti-abuse (per-user upload limits, comment velocity)
• Multi-tenant SaaS (per-tenant fairness)

When NOT to use it

• Fully internal APIs where all callers are trusted and instrumented — a circuit breaker is often better
• Ultra-low-latency paths (sub-millisecond) — the rate limiter adds overhead you may not want
• Fully static content — the CDN handles this

Failure modes

• Redis outage → local fallback allows unlimited traffic (better than blocking everything). Design for this.
• Legitimate spike gets throttled (e.g., a product launch) — always feature-flag rate limits so you can raise them fast
• Global rate limiter becomes a hot key — shard by hash prefix
• Bucket bookkeeping drifts across instances — use approximate token bucket with periodic sync
• Rate limit response body is expensive to generate — use a static 429 payload

Alternatives

• API Gateway with built-in rate limiting (Kong, AWS API Gateway) — cheapest
• Envoy / service mesh with rate-limit service (envoy-rls) — for internal East-West traffic
• Circuit breaker — different tool, complementary; circuit breaker protects YOU from downstream failures, rate limiter protects downstream FROM you
• Load shedding at the load balancer — coarse but works when you're already overloaded

Interview questions

• Compare token bucket, leaky bucket, and sliding window. When would you use each?
• Design a distributed rate limiter for 1M API keys at 1M RPS. Where's the state?
• Your rate-limit Redis is down. What happens to traffic?
• You want to allow bursts up to 100 requests but sustain only 10/sec per user. Which algorithm?
• How would you implement per-endpoint rate limits (different budgets for different APIs) efficiently?
• How does rate limiting interact with retries? What can go wrong?