api-gateway

API Gateway

Single entry point that handles auth, rate limiting, routing, and protocol translation for downstream services.

Why it exists

In a microservices world, you have dozens of backend services and one thing every request needs to do consistently: authenticate, authorize, rate-limit, log, trace, and route to the right service. If you do these in each service, you end up with N inconsistent implementations of the same logic. An API gateway centralizes cross-cutting concerns at the edge — so services stay focused on business logic.

How it works

The gateway sits between clients and services. On every incoming request it: validates the API key or JWT (authN), checks scopes/roles (authZ), enforces per-caller rate limits, does request/response transformation (protocol translation, response shaping), fans out to backend services (aggregation for BFF patterns), and adds observability headers (request ID, tracing context). Modern gateways (Kong, Envoy Gateway, AWS API Gateway) offer plugins so you don't have to fork the code to customize behavior.

Scaling characteristics

Stateless — scales horizontally like any service. Modern gateways handle 10K-100K RPS per instance. The bottleneck is usually the auth check (crypto verification of tokens) and the downstream fanout (if it's aggregating). Latency overhead is typically 1-5ms on cache hits; 20-50ms on JWT verification with an external identity check.

When to use it

You have multiple backend services and want consistent auth/rate limiting across all of them
You need protocol translation (e.g., REST → gRPC internally)
You want a BFF (Backend for Frontend) that aggregates several downstream calls
You need per-partner API keys with different rate budgets and pricing tiers
You need centralized observability (structured logs + traces per request)

When NOT to use it

You have one monolithic service — add a middleware chain instead
You have very high internal RPS between services (>500K) — a service mesh (Envoy sidecars) is often cheaper than routing everything through a central gateway
Your API is entirely internal to your VPC — DNS + IAM + service mesh may do it better

Failure modes

Gateway becomes a single point of failure — always run 2+ instances behind an LB
Auth service upstream goes down → gateway can't validate tokens → cache token validation with short TTL as a fallback
Bad rate-limit config accidentally blocks legitimate traffic — feature-flag rate limits so you can disable per-tier fast
Plugin/filter chain adds unbounded latency — set per-stage timeouts
Response aggregation timeout when one downstream is slow — set aggressive per-hop timeouts and return partial results

Alternatives

Service mesh (Istio, Linkerd) — auth/observability at every hop via sidecars; no central bottleneck but heavier operational model
Reverse proxy (Nginx, HAProxy) — cheap for routing + TLS, but you'll build auth/rate limiting yourself
Serverless per-endpoint (Lambda + API Gateway) — great for spiky, uneven load; expensive at sustained scale
GraphQL federation (Apollo Router) — if your API is GraphQL, the federation gateway subsumes the traditional gateway role

Interview questions

Where do you enforce auth — at the gateway, at each service, or both?
How do you handle a global rate limit across 10 gateway instances?
Your gateway is at 90% CPU because JWT verification is expensive. What now?
How would you canary a new gateway version without breaking clients?
The gateway needs a header from an upstream service (e.g., user tier). How do you get it?
What happens when a downstream service returns a 5xx to the gateway?

Systems that use this component

See how the real designs on this platform put api-gateway to work — concrete usage context per system.

Twitter/X timeline

intermediate

Kong-based gateway with rate limiting per user + per app

Open system

Instagram

intermediate

GraphQL gateway federating microservices

Open system

Payment System

intermediate

Rate limiting + API key management + request signing

Open system

Slack

intermediate

REST + WebSocket gateway with per-workspace rate limits

Open system

The story of edge routing

2013, Netflix: Zuul is born from a single question

In 2012, Netflix engineers had a problem: they'd migrated to AWS + microservices, and now there were hundreds of backend services. Every mobile app, every TV, every browser had to know which service to call for what. Auth logic was duplicated in 40 places. Rate limits were inconsistent. Logging was chaos. One question kept coming up in postmortems: "Where do we put the code that every request needs?"

Their answer, released as open source in 2013, was Zuul — a single JVM application that stood in front of every Netflix backend. Every request from every device hit Zuul first. Zuul verified auth, applied rate limits, added tracing, routed to the right service, retried on failure, and even did experimental traffic mirroring. Suddenly, cross-cutting concerns lived in one place. Netflix could ship new services in days without touching auth or telemetry code.

The pattern spread fast. Kong (2015) commercialized it on top of Nginx. AWS API Gateway (2015) built the serverless variant. Envoy (2016) came from Lyft with a mesh-native architecture. Today every non-trivial microservices deployment has a gateway — because the alternative is chaos multiplied by N services.

The core insight: a gateway isn't just a router. It's the policy plane for your entire API. Auth policy, rate policy, retry policy, observability policy — all defined once, enforced consistently, across every service. Get the gateway right, and every downstream service can focus on business logic.

What the gateway does in one request

1. Terminate TLS
2. Verify JWT / API key / OAuth token
3. Check rate limits (per-user, per-tier)
4. Apply routing rules → pick backend
5. Transform request (REST → gRPC?)
6. Forward with circuit breaker + retry
7. Transform response (shape / compress)
8. Log + emit trace

All in ~5ms overhead. That's the price of consistency, security, and observability across your entire API surface.

Historical timeline

2010
Amazon-internal service gateway
Amazon builds an internal API layer for its retail microservices. Team Ownership Boundary + Auth centralized. Pattern spreads inside AWS.
2011
Sam Newman coins "BFF"
Backend for Frontend pattern emerges at SoundCloud + ThoughtWorks — one gateway per client type (iOS, web, TV) instead of one giant "god" gateway.
2013
Netflix open-sources Zuul
First widely-used gateway pattern. JVM. Groovy filters. Powers >99% of Netflix traffic. Sets the industry template.
2015
Kong founded (on Nginx + Lua)
Marco Palladino + Augusto Marietti productize the gateway pattern. Plugin ecosystem. Enterprise + open source. IPO-track by 2022.
2015
AWS API Gateway launches
Serverless-native. Deep Lambda integration. Puts a gateway in reach of one-person startups. First truly managed gateway.
2016
Envoy released by Lyft
Matt Klein builds a C++ proxy from scratch for Lyft's microservices. Becomes the data-plane standard for service meshes.
2016
Istio + Linkerd
Service mesh emerges as the "gateway everywhere" pattern. Every service sidecar has gateway-like functions. Debate: mesh vs central gateway.
2017
Netflix retires Zuul 1, releases Zuul 2
Non-blocking Netty rewrite. Handles 100K RPS per instance. Fixes the blocking-thread bottleneck of Zuul 1.
2018
Apollo Federation for GraphQL
Apollo Router / Gateway becomes the "GraphQL gateway." Merges subgraph schemas, distributes queries, aggregates. New category.
2020
Cloudflare API Gateway launches
Gateway integrated with the CDN — auth + rate limit at the edge, before the request even reaches your infra. New economic model.
2022
Envoy Gateway launched
The Envoy community productizes a Kubernetes-native gateway on top of Envoy's proxy. Deep integration with Gateway API spec.
2024
AI gateways emerge
Portkey, LangDB, Kong AI Gateway — gateways specialized for LLM routing, cost tracking, prompt caching, model failover. New sub-category.

The 8-stage request lifecycle inside a gateway

Every request that touches your gateway goes through these stages in order. Understanding this pipeline is the difference between a gateway that's a black box and one you can debug at 3am. Click a stage or watch the walkthrough:

Auto-advancing every 2s

Stage 1: TLS terminate

Adds: 0.2ms

Decrypt HTTPS. Certificate pinning. Origin sees plain HTTP internally.

Watch for: Cert rotation, HSTS

Auth patterns: five ways the gateway proves who you are

Every gateway makes an authentication choice that affects latency, scalability, revocation, and security posture. Try each:

JWT (JSON Web Token)

Best for: User/session-based apps. Web + mobile.

How: Client sends token in Authorization header. Gateway verifies signature using public key (RSA/ECDSA) or shared secret (HMAC). Extracts claims (userId, roles, exp).

Latency added

0.5-2ms if key cached in gateway. No network hop needed.

Revocation

Hard — tokens are stateless. Common solutions: short TTLs (5-15min) + refresh tokens, or a revocation blocklist checked per request.

Pros: Stateless — gateway needs zero backend calls. Great for horizontal scale. Standard (RFC 7519). Libraries everywhere.

Cons: Can't revoke instantly. Large tokens (>2KB) waste bandwidth. Claims-in-token means you can't update permissions without new token.

When to reach for it: Standard web/mobile app auth. When gateway must scale without hitting a central auth service on every request.

BFF vs one-gateway-to-rule-them-all

There are two philosophies for the shape of your gateway tier. The choice shapes everything downstream:

Monolith gateway

One gateway serves all clients — iOS, Android, web, TV, partners.

iOS ─┐
Web ─┼→ Gateway ─→ Service A/B/C
TV ─┘

Pros: One config, one deploy, consistent policy. Simplest team model.

Cons: Every client sees every field. Payloads bloat. Impossible to optimize per-client.

Best for: Small teams. Similar clients. When client teams don't own their backend.

BFF (Backend for Frontend)

One gateway per client class. Each is a tiny aggregator tuned for one frontend.

iOS ─→ iOS BFF ─→ Service A/B/C
Web ─→ Web BFF ─→ Service A/B/C
TV ─→ TV BFF ─→ Service A/B/C

Pros: Each BFF is 10× smaller than a monolith. Client teams own their BFF. Payloads perfectly tuned.

Cons: N BFFs to deploy and secure. Auth/logging risk of drifting apart.

Best for: Large orgs. Distinct clients (mobile vs TV). SoundCloud/Netflix/Spotify pattern.

Modern hybrid: Cross-cutting policy (auth, rate limits) at a thin outer gateway; client-tuned aggregation in BFFs behind it. You get consistent security + client-optimized payloads. Netflix, Spotify, Uber all converged here.

Circuit breaker: how a gateway protects backends from itself

When a backend service is dying, sending it more traffic makes it worse. A circuit breaker at the gateway stops sending requests to a failing backend and returns an error immediately — protecting the backend and giving fast feedback to clients. Try the buttons:

Closed (healthy)

All requests pass through

Open (tripped)

All requests fail fast

Half-open

One probe request allowed

Current state (closed): All requests forwarded to backend. Failure counter reset on success.

Netflix Hystrix legacy: Netflix released Hystrix in 2012 — the first widely-adopted circuit breaker. Today Envoy, Kong, Istio all have it built in. Trip thresholds: 5-10 consecutive failures, or 50%+ error rate over a 10s window. Recovery: 30s-1min timeout before half-open probe.

Interactive: routing policy tester

The gateway's routing table is a decision tree. Try different requests and see which backend they hit:

Method

Path

Custom header (try: canary=true)

Try: /api/v1/orders/99, /graphql, /health, /api/v2/anything

Routed to

user-service

Matched rule: /^\/api\/v1\/users/

Canary rollout: 5% of matching requests go to the canary build. Traffic split done via consistent hash on user ID (so a given user consistently gets the same variant).

Path routing: most common. Match on URL prefix/regex.

Header routing: canary, A/B tests, region hints, mobile vs web.

Weighted routing: 95% v1, 5% v2 for gradual rollout. Same rule, weighted split.

Product comparison

Product	Model	Built on	Strength	Weakness
Kong	OSS + Enterprise	Nginx + Lua/Go	Massive plugin ecosystem, K8s-native	Lua learning curve, Enterprise pricing
AWS API Gateway	Fully managed	AWS proprietary	Zero infra, Lambda integration, cheap at low RPS	Vendor lock-in, cold-start latency, expensive at scale (>10K RPS)
Envoy Gateway	OSS (CNCF)	C++ (Envoy)	Fastest data plane, service mesh native, Kubernetes Gateway API	Config complexity, YAML-heavy
Google Apigee	Managed / on-prem	Java	Best-in-class API product management, monetization, dev portal	Expensive, GCP-centric, dated UI
Azure API Management	Managed	.NET	Azure integration, dev portal, policy DSL, monetization	Azure lock-in, cold-start on Consumption tier
Netflix Zuul 2	OSS	Java + Netty	Battle-tested at Netflix scale (100K+ RPS/inst), non-blocking	Groovy filters, JVM heavyweight, community activity lower
Traefik	OSS + Enterprise	Go	Auto-discovery (Docker, K8s), simple config, HTTPS-out-of-box	Fewer enterprise features than Kong, plugin ecosystem younger
KrakenD	OSS + Enterprise	Go	Aggregation/composition-first, ultra-low-mem, declarative config	Less known, smaller community
Cloudflare API Gateway	Managed (edge)	Rust + Workers	Runs at 300+ PoPs, integrated with WAF/DDoS, generous free tier	Vendor lock-in, complex pricing at high volume
Apollo Router	OSS + Cloud	Rust	GraphQL federation done right, subgraph composition, telemetry	GraphQL-only, less useful for REST/gRPC

Choosing quickly: Kubernetes-first team → Envoy Gateway or Kong. Fully-managed & on AWS → AWS API Gateway. Public developer API monetization → Apigee or Kong Enterprise. Edge-first (WAF + DDoS) → Cloudflare. GraphQL shop → Apollo Router. Small Go/Docker shop → Traefik.

12 real-world API Gateway patterns

Netflix

Zuul 2: 100 billion requests/day

Zuul 2 handles >100B requests/day across ~1000 gateway instances. Non-blocking Netty core. Filters written in Groovy (Kotlin now). Handles routing, auth, throttling, retry, and Chaos Monkey's traffic manipulation. Every device in the Netflix ecosystem passes through Zuul first.

Uber

Edge Gateway routes 10M RPS

Uber's Edge Gateway (custom Go) is the entry point for rides, Uber Eats, Freight. Per-request auth (mTLS internally, OAuth externally). ~10M RPS at peak. Includes DDoS mitigation, per-city rate limits, and rich request-header propagation for observability.

Stripe

Idempotency at the gateway

Every Stripe API request accepts Idempotency-Key. The gateway deduplicates retries by that key + a 24-hour Redis cache. If a client retries after a network glitch, the gateway returns the cached response instead of creating a duplicate charge. This is why Stripe integrations feel "just work" safe.

AWS API Gateway

Lambda authorizers for zero-infra auth

AWS lets you write a Lambda that returns an IAM policy per request. The gateway calls your Lambda, caches the result for 5min, and enforces the policy. Zero gateway configuration required — auth logic lives in one Lambda function. Used by countless startups running fully serverless.

Kong on Kubernetes

GitOps-driven gateway

Modern platform teams deploy Kong as a Kubernetes Ingress Controller. Routing rules, auth plugins, rate limits — all declared as YAML custom resources. Argo CD syncs them from Git. Changes go through PR review. Impossible to accidentally break routing without a code review.

Shopify

Ruby monolith → gateway-fronted mesh

Shopify's core is a Rails monolith. As they extracted services, they put an API Gateway (custom on Envoy) in front. Every extracted service is versioned, monitored, and rate-limited at the gateway. The monolith didn't have to change — the gateway made the migration invisible to clients.

Apigee at Walmart

B2B API monetization

Walmart uses Apigee to expose supplier + partner APIs. Dev portal for signup. Per-partner API keys with tiered quotas. Monthly billing reports pulled from Apigee analytics. This turns an internal API into a revenue stream.

Cloudflare

API Gateway at the edge

Cloudflare's API Gateway runs at 300+ PoPs. Auth, rate limits, schema validation — all executed at the edge, before requests ever leave the CDN. Origin servers see ~5% of traffic. Sub-10ms auth latency globally. WAF + bot management included.

Google Bank of Anthos

Istio + gateway reference

Google's Bank of Anthos reference architecture uses Istio's ingress gateway + service mesh for a bank-grade microservices demo. mTLS everywhere. Central policy definition. Distributed enforcement. Now a template for regulated industries.

Apollo GraphOS

Federation at Airbnb

Airbnb uses Apollo Router to federate ~50 subgraphs into one unified GraphQL schema. Frontend developers query one endpoint; the router splits into subgraph queries, aggregates, and returns. Auth + rate limits at the router. Backwards-compatible schema evolution via versioning.

Discord

Custom Elixir gateway for real-time

Discord fronts its WebSocket real-time API with a custom Elixir/Erlang gateway. Handles ~150M concurrent connections. Auth on connect, then long-lived socket. Reconnect logic + rate limits at the gateway. This is why Discord can support massive rooms without meltdown.

OpenAI ChatGPT

AI gateway pattern

OpenAI's edge is a Cloudflare-backed API Gateway with LLM-specific concerns: per-model rate limits, per-key cost tracking, prompt caching for repeated queries, model failover (GPT-4 → GPT-3.5 on error). New sub-category: "AI gateway" (Portkey, Kong AI Gateway) commercializing this.

Key takeaways

1A gateway is your policy plane — auth, rate limits, retries, observability all defined once, enforced everywhere.
2The 8-stage lifecycle (TLS → auth → rate limit → route → transform → forward → transform → log) is the mental model to debug any gateway.
3Pick the right auth pattern: JWT for stateless scale, OAuth for revocation, mTLS for zero-trust, API key for machine-to-machine.
4Circuit breakers at the gateway protect backends from cascading failures. Trip at 5-10 consecutive failures. Recover with a probe request.
5BFF pattern beats monolith gateway once you have >2 distinct client types. One gateway per client class, thin outer gateway for cross-cutting policy.
6Every gateway plugin adds latency. Aim for <5ms total overhead. If it's more, either move logic to backend or accept the trade-off consciously.
7For public APIs, add idempotency keys at the gateway. Turns retry-on-network-glitch into a safe, guaranteed-exactly-once operation.
8Modern gateway ≠ Zuul-in-a-box. Choose based on ops model: managed (AWS/Apigee), self-hosted (Kong/Envoy), edge-native (Cloudflare), mesh-native (Istio).

References & further reading

• Newman, S. (2015). Building Microservices. O'Reilly. Chapter on API gateways + BFF pattern is definitive.
• Netflix Tech Blog (2013). "Announcing Zuul: Edge Service in the Cloud." The original post that shaped the industry.
• Netflix Tech Blog (2018). "Open Sourcing Zuul 2" — the non-blocking Netty rewrite.
• Nygard, M. (2007). Release It!. Pragmatic. Coined "circuit breaker" as a systems pattern.
• Fowler, M. (2015). "Backends for Frontends." Blog post that popularized the BFF pattern.
• Klein, M. (2018). "Service Mesh Data Plane vs. Control Plane." Envoy's creator on the mesh vs gateway debate.
• Kong Docs: "How Kong Works." Excellent architecture walkthrough.
• Envoy Docs: "Life of a Request." Deep dive on the Envoy request lifecycle.
• Kubernetes Gateway API: The evolving standard for K8s ingress/gateway configuration.
• OAuth 2.1 RFC: The modern spec consolidating OAuth 2.0 + best practices.
• OWASP API Security Top 10: What every gateway should protect against.