Skip to main content
coordinator

Service Discovery (Consul, etcd, DNS-based)

How services find each other in a dynamic fleet — because IPs change, nodes come and go, and hardcoding is a losing game.

Why it exists

In a static environment, service-A calls service-B at 10.0.0.42:8080 — configured once. In a dynamic environment (containers, autoscaling, rolling deploys), IPs change constantly. A service discovery layer solves the question: 'what are the current healthy instances of service-B?' — with automatic registration on startup, deregistration on shutdown, and health checking. It's the plumbing that makes microservices possible.

How it works

Services self-register on startup by writing (name, IP, port) into the discovery store. Clients query the store to find current instances. A health check either polls each instance or receives heartbeats; unhealthy instances are removed. Some systems (Kubernetes) use DNS as the front — you look up 'service-b.default.svc.cluster.local' and get a list of current IPs. Others (Consul, etcd) expose a richer API for service metadata + tags.

Scaling characteristics

Discovery stores are read-heavy — millions of queries/sec is common. Write load (register/deregister/heartbeat) is bounded by fleet size. DNS-based discovery scales trivially via caching (TTL trade-off vs freshness). Consul/etcd scale to thousands of services; beyond that you shard by domain.

When to use it

  • Microservices architectures with autoscaling
  • Kubernetes-based deployments (service discovery is baked in via DNS + endpoints)
  • Service mesh setups (Consul + Envoy)
  • Multi-region deployments where routing needs to know regional availability
  • Any environment where instance IPs change (containers, spot instances)

When NOT to use it

  • Fully static, single-region, small fleet — a load balancer with hardcoded targets is simpler
  • When you can piggyback on the cloud provider's LB/DNS — often good enough

Failure modes

  • Stale registrations: a dead service is still returned to clients. Mitigate with aggressive health checks + short TTLs.
  • Split-brain in Consul during partition — some clients see one set of services, others another
  • Cascading heartbeat failures: discovery store overload causes it to mark healthy nodes as dead
  • DNS TTL mismatch: clients cache old IPs for minutes, seeing 'dead' services after failover
  • Client-side caching without invalidation causes routing to gone instances

Alternatives

  • Kubernetes Service + DNS — batteries-included for K8s workloads
  • Client-side load balancing with a service registry (Ribbon, gRPC name resolvers)
  • Envoy service mesh with xDS — dynamic config push from a control plane
  • Hardcoded LB + rolling deploys — works for stable environments; falls apart with autoscale

Interview questions

  • Compare DNS-based service discovery with a purpose-built registry (Consul/etcd). Pros and cons?
  • Your service dies but stays registered for 30s. What's the client-side impact and mitigation?
  • How does Kubernetes service discovery work under the hood (Endpoints, kube-proxy)?
  • Design a multi-region service discovery that keeps regional latency low.
  • What happens if the service discovery cluster itself goes down?
  • How would you route traffic to services during a rolling deployment without dropping requests?

Systems that use this component

See how the real designs on this platform put service-discovery to work — concrete usage context per system.

Netflix

intermediate

Eureka — battle-tested service discovery

Open system

Twitter/X timeline

intermediate

Custom Finagle-based service mesh

Open system

Instagram

intermediate

Envoy-based service mesh at scale

Open system
The story of finding services

Where did my service go? The problem microservices didn't know they had

For 30 years, service discovery was solved by DNS: your app hardcoded api.example.com, DNS returned an IP, done. Then microservices arrived. Suddenly you had 50 services × 20 instances each × ephemeral containers coming and going. IPs changed constantly. TTLs made DNS too slow. Static config broke overnight. The old solution stopped working.

The industry response was service registries — a live database of "who's alive where." Netflix Eureka (2012) let services register themselves + query for peers. HashiCorp Consul (2014) added health checks, KV, DNS interface. etcd and ZooKeeper served the same role for other stacks. Then Kubernetes (2015) baked service discovery into the platform — everything is a Service resource, backed by an in-cluster DNS + kube-proxy magic. You stopped even thinking about it.

Modern discovery is three layers: (1) DNS-based (still the most common, now with SRV records + short TTLs), (2) registry-based (Consul, Eureka, etcd — apps ask the registry directly), (3) service mesh (Istio, Linkerd — a sidecar proxy handles discovery + load balancing invisibly). Which you pick shapes everything else — the sidecars, the deployment tooling, the debugging.

The core insight: service discovery is the seam where infrastructure meets application code. Get it right, and adding a new service instance is invisible — traffic just starts flowing. Get it wrong, and you learn about a deployment failure only when 10% of requests start 500ing because DNS still points at the dead pod. The primitives are simple: health checks + registry + routing. Choosing the right composition is where design taste matters.

Three discovery models
DNS-based
Consul DNS, K8s CoreDNS, Route 53. Client resolves hostname to IP.
Registry-based
Eureka, Consul KV, etcd. Client queries API for peers.
Service mesh
Istio, Linkerd sidecars. Discovery happens in the proxy — app is oblivious.
Modern default: K8s + service mesh. Discovery is invisible to app code. Sidecars do the work.

Historical timeline

  1. 1983
    DNS invented (RFC 882/883)
    Paul Mockapetris designs DNS. Solves 30 years of hostname discovery in one protocol.
  2. 1996
    DNS SRV records (RFC 2052)
    SRV records add port + priority discovery. Foundation for many service discovery schemes.
  3. 2007
    mDNS / Bonjour
    Apple standardizes zero-config service discovery on local networks. AirPrint, AirPlay use this.
  4. 2008
    Apache ZooKeeper
    Yahoo builds coordination service. Used for both locks + service registry in Hadoop stacks.
  5. 2012
    Netflix Eureka
    Netflix open-sources Eureka — service registry + client discovery. Companion to Ribbon (client-side load balancing).
  6. 2013
    Consul + etcd emerge
    HashiCorp Consul (Sept 2013) + CoreOS etcd (Aug 2013). Both aim at coordinated service discovery + KV.
  7. 2015
    Kubernetes 1.0
    K8s ships with in-cluster DNS + Service abstraction. Service discovery becomes a platform primitive.
  8. 2016
    Envoy proxy by Lyft
    Envoy's xDS APIs enable dynamic service discovery for sidecars. Foundation for service meshes.
  9. 2017
    Istio + Linkerd 2.0
    Service mesh becomes mainstream. Discovery, routing, retries, mTLS — all in the sidecar.
  10. 2018
    AWS App Mesh + Cloud Map
    AWS enters service discovery. Cloud Map for registry. App Mesh for mesh routing.
  11. 2019
    Consul Connect adds mesh features
    HashiCorp evolves Consul into a full mesh with mTLS + policy. Bridge between registry + mesh worlds.
  12. 2020
    Kubernetes EndpointSlices
    K8s scales endpoint discovery. Old Endpoints objects broke at 1000+ pods; slices scale linearly.
  13. 2022
    Ambient mesh (Istio)
    Sidecar-less service mesh emerges. Discovery + routing without a proxy per pod.
  14. 2024
    Discovery gets AI-aware
    Service registries add metadata for AI-model versioning, routing, capacity — LLM inference nodes register per model.

Four discovery patterns — which shape fits your architecture?

Choosing between these shapes what your infrastructure looks like. There is no universal winner — it's about where you want to put the complexity:

Client-side discovery

Client                    Registry
  │                          │
  ├─── who is api? ──────────►│
  │◄── [ip1, ip2, ip3] ──────┤
  │
  ├─── request → ip2 (client picks)
Pros: Client owns load balancing. No extra hop. Full control over routing.
Cons: Client SDK required in every language. Each app must implement retry/LB logic.
Use: Java-centric stacks (Eureka + Ribbon). Netflix pioneered this.
Products: Netflix Eureka + Ribbon, Consul client SDKs.

Health checks — what "alive" actually means

A service is alive when it can serve requests correctly. This is harder than it sounds. "Ping succeeds" is not the same as "DB queries work." Modern discovery systems support multiple health check types:

Liveness probe
GET /health/live → 200 if the process is running (not necessarily useful)
Status: OK
Action: restart the pod if failing.
Readiness probe
GET /health/ready → 200 if the service can accept traffic (deps warm)
Status: READY
Action: registry stops routing traffic if failing.
Startup probe
GET /health/startup → 200 once cold init is done
Only used during pod boot. Prevents liveness kills during slow starts.
Deep health check
GET /health/deep → validates DB, cache, downstream services
Status: OK
Warning: don't make this the readiness check — cascading failures if a downstream dies.
Common trap: tying readiness to downstream DB. If your DB has a hiccup, ALL your service instances mark themselves not-ready → discovery removes them all → outage. Instead: use liveness for "process alive," readiness for "ready to serve THIS request," and monitor deep-health separately.

Two ways for services to register themselves

Once you have a registry, the next question is: who tells the registry about new instances?Two philosophies, both viable:

Self-registration

Service registers itself on startup, deregisters on shutdown. Sends heartbeats.

on_start:
  registry.register(self, tags=[v2, primary])
  every 10s: registry.heartbeat()
on_shutdown:
  registry.deregister(self)
Pros: Simple, no external orchestrator. Service knows its own health.
Cons: Every service needs the SDK. Zombies if crashed instance can't deregister.
Products: Netflix Eureka, Consul agent-based, ZooKeeper Curator recipes.

Third-party registration

A separate registrar (Kubernetes, ECS, Nomad) watches for new instances and populates the registry.

K8s Pod created → controller sees →
  populates Endpoints/EndpointSlices →
  CoreDNS updates → clients discover
Pod dies → Endpoints cleaned →
  discovery converges in seconds
Pros: Apps don't need registry knowledge. Clean cleanup on pod termination.
Cons: Requires an orchestrator. Some info (like "warming up") still needs app cooperation.
Products: Kubernetes (Endpoints), AWS ECS Service Discovery, Nomad, HashiCorp Consul auto-registration.
Modern reality: most K8s deployments use third-party registration (Kubernetes itself does it). Eureka self-registration was Netflix's pre-K8s pattern. Consul supports both.

The Kubernetes model — discovery baked into the platform

Kubernetes made service discovery so smooth that many devs never think about it. Understanding the layers helps when something breaks:

1. Pod runs with labels
labels: app=api, version=v2
2. Service selects pods by label
Service "api" matches pods with app=api → creates virtual IP
3. EndpointSlices track matching pods
EndpointSlices auto-update: [10.0.1.5, 10.0.2.8, 10.0.3.11]
4. CoreDNS resolves service name
api.default.svc.cluster.local → Service ClusterIP
5. kube-proxy load-balances to a pod
Uses iptables (or IPVS or eBPF) to DNAT → random pod IP
Zero app config: your code calls http://api/ — everything else is Kubernetes magic.
Failures debugged in order: pod healthy? → Service selector match? → EndpointSlices populated? → CoreDNS resolving? → kube-proxy rules present?
Modern twist: service meshes replace kube-proxy's dumb round-robin with smart routing (retries, circuit breaking, canary).

Product comparison

ProductModelStrengthWeakness
Kubernetes Service + CoreDNSOSSBaked into K8s. Zero-config for pods in-cluster. Standard.Only works in K8s. External clients need Ingress or LB.
HashiCorp ConsulOSS + EnterpriseMulti-DC. Rich health checks. KV + DNS interface. Now also mesh.License changed (2023). Extra cluster to run outside K8s.
Netflix EurekaOSS (Netflix)Battle-tested at Netflix. Client-side LB with Ribbon.Java-centric. Eureka 2.x abandoned. Netflix moved to K8s.
etcdOSS (CNCF)Raft-based. Powers K8s control plane. Reliable.Low-level API. Requires you to build discovery on top.
Apache ZooKeeperOSSMature. Used by Kafka, HBase, Hadoop. Rich primitives.JVM ops. Session complexity. Older API.
AWS Cloud MapManagedAWS-native. Integrates with ECS, EKS, Lambda. DNS interface.AWS-only. Costs add up at scale.
IstioOSS (CNCF)Full mesh: discovery + mTLS + traffic mgmt + observability.Complex. Envoy sidecars add resource overhead. Steep learning curve.
LinkerdOSS (CNCF)Simpler than Istio. Rust-based sidecar. Fast + light.Fewer features than Istio. Smaller community.
AWS App MeshManagedAWS-managed Envoy mesh. Deep AWS integration.AWS-only. Recently declared legacy (VMware Tanzu status).
SkyDNS + others (legacy)OSSPredecessor to CoreDNS in K8s. Historical value.Obsolete.
Google Cloud Service DirectoryManagedGCP-native. Deep GCP integration.GCP-only.
Envoy xDS ecosystemOSSConfig protocol that many meshes speak. Standardizes discovery integration.Protocol, not a full product. Requires a control plane.

How to choose: K8s shop → CoreDNS + Services. Need multi-DC → Consul. AWS-managed → Cloud Map. Full mesh → Istio (rich features) or Linkerd (simplicity). Java + old stack → Eureka. Kafka-adjacent → ZooKeeper.

12 real-world service discovery deployments

Netflix

Eureka + Ribbon for a decade

For 10+ years, Netflix used Eureka (registry) + Ribbon (client-side LB) to discover services across thousands of instances. Client-side LB meant no central bottleneck. Recently migrated much of this to K8s + service mesh.

Google Kubernetes

CoreDNS at >5M queries/sec

Google Cloud runs Kubernetes clusters with CoreDNS serving hundreds of thousands of pods. Every intra-cluster call resolves through CoreDNS. Zero manual config: pod → Service → EndpointSlices → CoreDNS → pod IP.

Uber

Custom RPC discovery via TChannel

Uber's TChannel RPC framework (predecessor to gRPC) included baked-in service discovery via Ringpop consistent hashing. Every service instance participated in a ring. Now migrating to gRPC + mesh.

HashiCorp

Consul at Roblox scale

Roblox uses Consul for service discovery across ~13K servers. Every game server, matchmaker, chat service registers. Consul's multi-DC support key for regional deployments.

Airbnb

SmartStack (Nerve + Synapse) legacy

Before K8s, Airbnb ran SmartStack — Nerve registered services in ZK, Synapse (a local HAProxy) proxied to them. Predecessor to modern service mesh, retired for K8s + Envoy.

Twitter

Finagle + ZooKeeper

Twitter's Scala RPC framework Finagle uses ZooKeeper for service discovery. Every service registers in ZK; clients query ZK for peers. Now blending with mesh approaches.

Lyft

Envoy + xDS

Lyft built Envoy and pioneered the xDS discovery protocol — sidecars dynamically discover clusters + endpoints from a control plane. Today the standard mesh control-plane interface.

AWS

Cloud Map + App Mesh + ECS

AWS integrates Cloud Map (registry) with ECS (auto-registration) and App Mesh (mesh routing). One consistent story across compute types.

Istio at Salesforce

Massive mesh deployment

Salesforce runs Istio across many production clusters. Mesh handles discovery, mTLS, retry, canary rollouts. Multi-cluster mesh for cross-region traffic.

Linkerd at HP

Simpler mesh choice

HP chose Linkerd over Istio for their internal platform. Simpler ops, smaller sidecar footprint. Trade: fewer features.

SkyScanner

Consul + Cassandra + K8s hybrid

SkyScanner runs a mix: Consul for legacy VMs, K8s Services for new microservices. Consul-K8s connector bridges them. Migration paths matter for real orgs.

GitHub Codespaces

Custom internal discovery

GitHub Codespaces (which spin up VMs on demand) built custom service discovery on Envoy + Redis. Every codespace VM registers itself with its assigned user + capacity. Serves millions of ephemeral instances.

Key takeaways

  • 1Service discovery has three main models: DNS-based, registry-based, or service mesh. Each has a place.
  • 2Modern default: Kubernetes + CoreDNS + Services. Zero app config for in-cluster discovery.
  • 3Health checks matter: separate liveness ("process alive") from readiness ("ready to serve") — DON'T check downstream DB from readiness.
  • 4Two registration models: self-registration (Eureka style) vs third-party (K8s style). K8s is now dominant.
  • 5Service mesh (Istio, Linkerd, Consul Connect) offloads discovery + retry + mTLS to sidecars. Adds ops overhead but reduces app-code complexity.
  • 6For multi-DC: Consul. For K8s: built-in Services. For AWS: Cloud Map. Java monolith legacy: Eureka.
  • 7Watch out for DNS caching. Language-level caches (JVM by default) can hold IPs for minutes — cause outages after failovers.
  • 8Discovery is the seam between infra and code. Get the abstraction right and adding services is trivial. Get it wrong and every deploy is a debugging session.

References & further reading

  • • Mockapetris, P. (1983). RFC 882/883. The DNS specification.
  • • Gulbrandsen, A. et al. (2000). RFC 2782. DNS SRV records.
  • • Netflix Tech Blog: "Netflix Shares Cloud Load Balancing And Failover Tool: Eureka!" (2012). The introduction.
  • • HashiCorp Consul Docs: "Service Discovery." Excellent conceptual overview.
  • • Kubernetes Docs: "Service" + "EndpointSlices." The canonical explanation of K8s discovery.
  • • Klein, M. (2017). "Service mesh data plane vs. control plane." Envoy founder's explanation.
  • • Istio Docs: "Traffic Management" section. The definitive service mesh reference.
  • • Newman, S. (2015). Building Microservices. Chapter on service discovery is excellent.
  • • Kleppmann, M. (2017). DDIA, chapters on networking + partial failure.
  • • CNCF Landscape: filter for "service proxy" and "service mesh" — best overview of the ecosystem.