Pods come up. Pods go down. IPs get recycled. Rolling deploys replace every instance. In a dynamic fleet, hardcoded addresses stop working in seconds. The pattern that makes "which instance handles this request?" a solved problem — and stays solved as the fleet changes shape underneath.
A microservice architecture in the era before Kubernetes generally assumed the world was static. You knew where each service lived; you wrote the address in a config file; you occasionally updated the config when things moved. That world does not exist any more. A modern service pod comes up with an IP the scheduler picked at random; it lives until a deploy, an autoscale event, a node failure, a spot-instance eviction, or a rolling upgrade replaces it; then it's gone, and the IP goes back into the pool to be handed out to something completely unrelated. The pod's identity was never its IP — it was its role. "The pod handling payment requests in this cluster" is stable; "10.42.3.17" is not. Service discovery is the indirection that makes this obvious-in-hindsight distinction operationally usable at fleet scale.
payment-svc-6d4c9-x8h. Kubelet assigns IP 10.42.3.17 from the node's CIDR pool. Pod goes into Pending → ContainerCreating. // EVENT: pod created · IP: 10.42.3.17Endpoints object for the payment-service. Traffic starts flowing. // EVENT: ready · 12s from schedule to servingSIGTERM; pod enters Terminating state. Endpoints removed from the Service's ready list. // EVENT: eviction · 12 min uptime10.42.3.17 returned to the node's pool for reuse. The identity that lived here for 12 minutes is gone forever. // EVENT: reclaimed · IP goes back to poolpayment-svc-9d1b8-k2m from the new image. Kubelet happens to assign IP 10.42.3.17 — the same IP the old pod had. Same address, different code, different config, different identity. // EVENT: IP recycled · zero warningAny caller with the 10.42.3.17 address hardcoded got: working service (09:00-09:12) → connection refused (09:12-09:13) → completely different service (09:13+). And this is one pod in a fleet that might see this happen 200 times a day across every pod under normal operations. The IP was never the identity — the role was. Service discovery is the layer that lets callers ask for the role and get whatever set of IPs currently fulfills it, however the fleet has been rearranged.
The temptation to solve this at the operations layer — "just always tell the caller the new IPs" — is what config-management systems (Chef, Ansible, Puppet) attempted for a decade with limited success. It doesn't scale for a simple reason: the rate of change in a modern fleet is faster than the rate at which humans can push config. Kubernetes reschedules pods on the order of seconds. Autoscalers trigger every minute. Node failures happen without warning. A rolling deploy of a 50-pod service completes in ~10 minutes. Any config-push mechanism that requires human-in-the-loop or even human-scheduled distribution runs orders of magnitude slower than the events it's trying to keep up with. The service graph is changing shape faster than static configuration can describe it. This is why every serious microservices platform ships with an integrated discovery mechanism, and why "how do you discover services?" is one of the first architecture decisions that has to be made.
The good news is that the abstraction is the same in every implementation: give the caller a stable name (payment-service, or payment.default.svc.cluster.local), let the runtime resolve that name to whichever instances are currently healthy, and hide the volatility of the underlying IPs entirely. The name is what the developer writes; the resolution is what infrastructure does. Where implementations differ is where the resolution happens — does the caller itself pick an instance (client-side discovery, §02), or does the caller talk to a stable virtual address that fronts the fleet (server-side discovery, §02)? — and where the registry lives (§03). The choices there drive the operational tradeoffs the lab in §04 lets you feel.
Given that discovery is an indirection, the architectural question is simply where does the indirection live. Two patterns dominate. In client-side discovery, the caller itself queries a registry, receives the current list of healthy instances, and picks one to talk to — the intelligence lives in the client. In server-side discovery, the caller talks to a well-known virtual address that fronts the fleet, and a load balancer or proxy behind that address handles instance selection — the intelligence lives in the infrastructure. Both patterns solve the same problem; they trade different things for different failure modes. The Kubernetes world has settled decisively on server-side discovery (via Service objects and kube-proxy); the classical Netflix stack (Eureka + Ribbon) is the canonical example of client-side. Both patterns are in production somewhere today, and understanding when each fits is more useful than memorizing which one Google uses this year.
The caller queries a registry, receives the current list of healthy instances, and load-balances itself. Netflix Eureka + Ribbon was the canonical implementation; each service was a Eureka client that fetched the registry, cached it, and applied its own load-balancing algorithm (round-robin, weighted, zone-aware). Every service does routing decisions in its own process. Modern equivalents include HashiCorp Consul with connect-native SDKs, and some gRPC name-resolver setups.
Trade: smart clients + registry access from every service = library integration in every language. Zone-aware routing, custom load-balancing policies, and sophisticated failure handling are natural fits. But the polyglot library problem from M.34 applies — different languages, different Eureka client behaviors, different cache staleness semantics.
The caller talks to a stable virtual address (DNS name or VIP); a load balancer or proxy behind that address picks the actual instance. Kubernetes Services with kube-proxy is the dominant example — payment.default.svc.cluster.local resolves to a virtual IP; iptables (or IPVS, or eBPF via Cilium) DNATs the connection to a randomly-chosen healthy pod. Caller doesn't know the pods exist.
Trade: dumb clients — just DNS + TCP, works in every language identically. But sophisticated routing (zone-aware, custom LB policies, latency-based) requires additional infrastructure (a service mesh from M.34). The mechanism becomes opaque to the caller, which is usually a feature but occasionally hides useful information.
Two things drive the modern preference for server-side discovery. First, Kubernetes standardized it: every workload deployed on k8s gets a Service object almost for free, and the DNS surface is uniform across languages. Second, service meshes (M.34) took over the sophisticated-routing use cases that used to justify client-side discovery: zone-aware routing, weighted traffic shifting, custom load-balancing policies, and locality preference are all now mesh sidecar concerns rather than in-app library concerns. This has left client-side discovery in a shrinking niche — teams that either can't run Kubernetes (rare and getting rarer), or that made the Eureka bet 8+ years ago and haven't migrated. New systems in 2025 almost universally use server-side discovery, integrating with a mesh when they need routing sophistication beyond what plain kube-proxy provides.
A subtle point worth naming: server-side discovery doesn't eliminate the registry — it hides it behind a DNS name. Kubernetes still has a registry (the Endpoints object, updated by the endpoint controller watching pod lifecycle events, persisted in etcd, watched by kube-proxy on every node). The developer just doesn't interact with it directly. When people say "we don't have a service registry, we just use Kubernetes DNS," what they actually mean is "the registry is managed by the platform, and we consume it through a standardized DNS interface." The registry is still there; the abstraction just moves the interaction point. This matters because the registry's failure modes (§05) still apply — they just show up as "kube-proxy having stale endpoints" instead of "Eureka cache is stale."
Behind every discovery mechanism is a service registry — a database that answers "which instances of service X are currently healthy?" The three interesting design questions are always: how does the registry find out about instance state? (health check model), how consistent does its view need to be? (CAP tradeoff), and what does it do when the network makes those hard? (failure mode). Different registries make different choices; the choices show up in exactly the situations you don't want to test in production. Every registry that's ever run in a real cluster has stories about the day its consistency model met a partition or its health-check model met a network hiccup.
The de-facto standard for cloud-native services. Pods self-register via kubelet health reports to the API server; the endpoints controller aggregates ready pods into Endpoints objects; kube-proxy on every node watches those and programs iptables/IPVS/eBPF rules to route the Service VIP to actual pod IPs. CoreDNS resolves service names to VIPs. The whole thing is invisible when it works.
Health via readiness probes (HTTP GET, TCP connect, exec command) run by kubelet locally on each node. Failing readiness removes the pod from Endpoints. Pull-based, node-local, no separate health-check service.
CP · etcd is strongly consistentA standalone service registry with strong consistency (Raft-backed). Services register themselves via HTTP API or config file; Consul agents on each node health-check them (HTTP, TCP, script, TTL) and propagate state through the Raft cluster. DNS interface (<service>.service.consul) plus HTTP API for consumers. The most common non-Kubernetes registry.
Works cross-cluster and cross-datacenter — one of Consul's genuine strengths. WAN gossip pool federates multiple clusters into a single logical service catalog. Kubernetes DNS ends at the cluster boundary; Consul crosses it.
CP · Raft consensus, prefers correctnessThe original AP-style discovery service, designed for the specific reality of running thousands of services across AWS availability zones. Client-side discovery model (from §02) with sophisticated Ribbon load-balancing. Explicitly chose availability over consistency — a stale-but-available Eureka is better than a fresh-but-partitioned one when your callers just want to talk to something.
Famous "self-preservation mode": if too many instances drop out in a short window, Eureka assumes network partition (not mass failure) and stops evicting. Prevents the worst failure mode: everyone marked unhealthy simultaneously.
AP · available even when staleNot really registries themselves — the primitives you use to build one. etcd (Raft) backs Kubernetes. ZooKeeper (Zab consensus) historically backed a lot of Apache-ecosystem discovery (Curator, Kafka's original controller). Both provide strongly consistent, watchable key-value stores that a discovery layer can be built on top of. You don't usually run these standalone for discovery any more.
Failure mode: consensus loss (loss of quorum) stops writes entirely, including new registrations. Chose consistency at the cost of "no new services can register while we're partitioned."
CP · consensus-backed, no split brainThe CAP tradeoff (M.21 callback) matters more here than almost anywhere else in a system. During a partition, a CP registry either stops serving requests (loss of quorum → no reads or writes) or serves stale data with strong guarantees about that staleness. An AP registry keeps serving from whatever node the caller can reach, accepting that different callers may see different snapshots. Which choice is right depends on what breaks worse — being unable to discover services, or discovering some services that don't exist any more. For most workloads, the answer is that being unable to discover anything is catastrophic (nothing can talk to anything), while calling a slightly-stale endpoint list is recoverable (retries + timeouts handle it). This is why Eureka's explicit AP choice — controversial in 2012 when Netflix published about it — has aged well. Discovery is the classic case where CAP theorem's abstract tradeoff meets a concrete "what fails worse?" question, and AP usually wins.
The other decision every registry has to make is the health-check model: does the registry push (instances heartbeat in, registry evicts stale entries) or pull (registry actively probes instances)? Push scales better (registry has O(1) work per instance regardless of instance count) but depends on instances being cooperative — a wedged process that stops heartbeating gets evicted, but a stuck one that still heartbeats but can't serve traffic looks healthy. Pull catches more failure modes (registry does its own probing) but scales worse (registry does O(N) work) and creates a bottleneck at large fleet sizes. Kubernetes solved this by making the health checks node-local: each kubelet checks its own pods, and only reports the summary to the API server. Distributed pull, effectively. This scaled to tens of thousands of pods per cluster because the health-check work distributed across the same nodes that were running the workload. It's a subtle piece of the k8s design that only becomes obvious when you compare it to the earlier centralized approaches.
Below: a 5-instance payment-service fleet, called by an order-service. Pick an architecture — hardcoded IPs (the naive baseline), client-side discovery (Eureka-style, caller queries registry then picks), server-side discovery (Kubernetes DNS + kube-proxy). Then pick a scenario — steady state, mid-rolling-deploy, registry outage, or a flapping unhealthy pod — and see how each architecture handles it. Watch the request success rate, the staleness window, and where things break when the registry itself is degraded.
Service discovery has an unusual property in a running system: it's completely invisible when it works. Developers write http://payment-service/api/charge, requests succeed, nobody thinks about it. That invisibility is exactly why the failure modes are surprising when they show up — nobody remembers that there was a discovery layer to break in the first place. The five patterns below cover most of the discovery-layer incidents you'll see in a mature system. Recognizing them fast is worth an outage or two of experience; recognizing them from the design phase is worth much more.
Every discovery mechanism has a cache somewhere — kube-proxy's iptables rules, an Eureka client's local snapshot, a DNS resolver's TTL. When a pod dies faster than the cache updates, calls to it get connection refused. Symptom: intermittent errors that clear up in 30-60 seconds. The fix: retries at the caller (mesh or app-level) to survive stale-cache windows without user impact. The prevention: understanding your discovery mechanism's staleness bound before you rely on it.
A pod near a resource limit oscillates between "healthy enough to respond to the probe" and "too overloaded to serve real requests." Its rapid transitions in and out of the registry create a stream of "pod removed / pod added" events that ripple through every caller. The fix: tune probe periods and failure/success thresholds (Kubernetes' initialDelaySeconds, periodSeconds, failureThreshold) so transient blips don't cause registry churn. Combine with pod-level resource requests that keep the pod stable.
A network partition inside a distributed registry (Consul, Eureka, or the Kubernetes control plane itself) can produce inconsistent views across callers. Some services succeed; others get connection errors; the diagnosis takes hours because the state on either side of the partition looks locally consistent. The fix: CP registries stop serving on quorum loss (preventing split-brain but causing outage); AP registries accept temporary inconsistency (Eureka's model). Choose deliberately; monitor for the anomalous state.
A registry restart or leader-election event triggers every client in the cluster to re-register or re-fetch state at the same moment. A 10,000-pod cluster fires 10,000 requests at the registry in the first second after recovery, and the registry falls over immediately. The fix: exponential backoff with jitter in every client (M.34 mesh policies help here); rate-limiting on the registry's write path; staggered reconnection with client-side sleep-and-retry. The specific defense: never assume the recovery path can handle N-times the steady-state load.
Kubernetes readiness probes are polled every few seconds. A pod that came up 100ms ago hasn't been probed yet, so its state defaults to "not ready" — but a race condition or a misconfigured initialDelaySeconds can cause it to be added to Endpoints before it's warmed up. First-request-after-cold-start latency spikes into hundreds of milliseconds because JIT compilation, connection pool warmup, or cache warming is still happening. The fix: proper readiness probes that check the actual serve path, not just /healthz. Startup probes (a Kubernetes feature) for slow-starting services. Understand what "ready" actually means for your service.
The composite lesson: service discovery is a distributed system's memory of itself, and it has all the failure modes that memory of distributed state always has — cache staleness, split brain, thundering herd on recovery, races between "is ready" and "is discoverable." These aren't discovery-specific bugs; they're distributed-systems bugs that show up at the discovery layer. That's why the same intuitions from M.21 (CAP), M.22 (consistency models), and M.24 (consensus) transfer here directly. If you know why Raft trades availability for consistency during partition, you already know why Consul's discovery stops serving during quorum loss. If you understand vector clocks and eventual consistency (M.22), you understand why Eureka's stale-but-available data model works despite feeling wrong the first time you see it. The infrastructure changes; the underlying tradeoffs don't.
The tools people converge on reflect this. Kubernetes' built-in discovery is dominant for anything running on k8s — the Service + Endpoints + kube-proxy + CoreDNS pipeline is invisible, cheap, and usually correct. Consul retains a role in hybrid or non-Kubernetes environments, and for cross-cluster/cross-datacenter federation (a genuine k8s gap). Eureka is legacy in most contexts now — Netflix themselves moved to a mesh-based approach, and the "client-side discovery with smart libraries" pattern lost to "server-side discovery through the mesh." etcd and ZooKeeper are foundational and still used, but almost never as user-facing discovery — they're the primitives underneath. The dominant 2025 pattern is Kubernetes DNS augmented by a service mesh for anything more sophisticated than round-robin routing, with Consul the answer for the cases k8s can't reach.
The terms every Kubernetes Service YAML, every Consul config, every "our discovery is broken" postmortem assumes you already know.
Endpoints in etcd). Updated by health-check results.Endpoints object. Slices the endpoint list into chunks to reduce the size of watch updates when large services change. The scaling fix that let single Services span 10,000+ pods.Test the discovery intuition. Click an answer; explanation drops in instantly.
Perfect. Client-side vs server-side, the registry's CAP tradeoff, failure modes — all yours. Now: once a request has traveled through the gateway, the mesh, and been discovered to the right pod, how do you actually see what happened to it end-to-end? That's distributed tracing.
The invisible-when-it-works infrastructure that makes "call this service" survive constant pod turnover.
In a dynamic fleet, pod IPs recycle every restart. Hardcoded addresses become stale in seconds. The role — "the pod handling payment requests" — is stable; the address serving that role is not. Service discovery is the layer that lets callers name roles instead of addresses.
The intelligence lives in the caller (Eureka+Ribbon: caller queries registry, picks instance) or in infrastructure (K8s Service + kube-proxy: caller uses a name, infra picks). Server-side won because Kubernetes standardized it and meshes absorbed the sophisticated-routing use cases.
The registry is a database with an urgency problem. Eureka chose AP; Consul and etcd chose CP. Both were right — for different questions about what fails worse: being unable to discover anything, or discovering something stale. Same tradeoffs as Phase E, different layer.