Service Discovery and Configuration Management Questions

Direct answer

Run one independent registry per datacenter (never one consensus cluster stretched across datacenters), make every lookup local-first, and fail over to the nearest healthy datacenter through an explicit, per-service failover policy. On top of that, choose the lookup mechanism by client type: DNS as the universal baseline, client-side discovery from the registry for services that need fast, health-aware balancing, and a service mesh where you also need mutual TLS (both sides of a connection proving their identity with certificates, not just the server) and fine-grained traffic control. For a typical platform I would commit to: Consul (or etcd-backed equivalent) per datacenter, federated (each datacenter's registry stays independent but can answer queries about the others through a lightweight cross-datacenter link); its DNS interface for everyone; mesh adopted per service where the security or routing need justifies the extra moving parts.

Terms first

• Service registry: the live list of healthy instances per service (Consul, etcd, ZooKeeper, or Kubernetes' own API).
• Client-side discovery: the calling service queries the registry itself (usually through a library) and picks an instance.
• Server-side / DNS discovery: the caller just resolves a name; something else (DNS server, load balancer) decides.
• Service mesh: a proxy sidecar next to every instance does discovery, load balancing, retries and encryption, fed by a control plane.
• Raft quorum: registries like Consul and etcd agree on writes using the Raft protocol, a consensus algorithm in which one elected leader orders every write and a majority of servers must acknowledge it before it is durable.
• Failover: sending traffic elsewhere when the preferred instances are unhealthy.
• Watch: a long-held request a client keeps open against the registry; instead of the client re-asking on a timer, the registry pushes each change down that open connection the moment it happens.
• Outlier detection (ejection): a proxy watches the real responses a backend returns and temporarily stops sending it traffic after enough of them look like failures, without waiting for a separate health check to catch up.
• Locality / locality-aware failover: preferring an endpoint in the caller's own datacenter first, and reaching into another one only when nothing local is healthy; "locality-weighted" means splitting traffic across localities by a configured ratio rather than all-or-nothing.

Why not one global registry

A Raft cluster commits a write only after a majority acknowledges it. Spread 5 servers across 3 datacenters that are 70 ms apart, a typical 2-2-1 split, and no single datacenter holds the 3-of-5 majority a write needs on its own, so every registration, deregistration and health update pays at least one cross-datacenter round trip (about 70 ms) before it commits, and a network partition can leave the minority side (whichever datacenters are left with fewer than 3 reachable servers) unable to register anything at all. Keeping each datacenter's registry local means local writes commit in about a millisecond and a partition leaves every datacenter fully functional on its own. Consul's design follows this: servers form a Raft cluster per datacenter, and datacenters are joined by a lighter WAN (wide area network) gossip layer (nodes periodically exchange state with a few random peers instead of every node talking to every other one, so it stays cheap even at scale) used for cross-datacenter queries, not for shared consensus.

Comparison

Worked example: failover time after an instance dies

Assume health checks every 10 s, marked critical after 3 failures, DNS TTL of 30 s.

• DNS: detection 3 x 10 = 30 s, then clients may keep the old answer for up to 30 s more: worst case about 60 s of some traffic still going to the dead instance (longer if a client caches past the TTL).
• Client-side with watch: detection 30 s, then the watch delivers the change in well under a second: about 30 s, and a library that retries on connection failure hides most of that window.
• Mesh: same 30 s for the registry path, but the proxy also does outlier detection on live traffic (Envoy, a widely used sidecar proxy, ejects a host by default after 5 consecutive 5xx errors, the HTTP status codes that signal a server-side failure), so a hard-down instance stops receiving requests after a handful of failed requests, often before the health check notices.

The recommended design

flowchart TB
  subgraph DC1[Datacenter 1]
    R1[Registry cluster, 5 servers] --- A1[Services: DNS or library or sidecar]
  end
  subgraph DC2[Datacenter 2]
    R2[Registry cluster, 5 servers] --- A2[Services: DNS or library or sidecar]
  end
  R1 <-->|WAN federation: cross-DC queries only| R2
  A1 -->|failover only when local instances unhealthy| A2

1. Local-first resolution. payments.service.consul (or the mesh equivalent) returns only instances in the caller's datacenter by default.
2. Explicit failover per service. In Consul this is a prepared query with a failover list ("nearest 2 datacenters if no local healthy instances"); in a mesh it is locality-weighted load balancing. Opt-in per service, because failing over a service whose database is local can make things worse (cross-DC latency on every query, or writes against a read-only replica).
3. Mechanism by client type. Legacy and third-party software uses DNS. Latency-sensitive internal services use the client library or the mesh. Services needing zero-trust networking (verifying every call cryptographically instead of trusting it just because it came from inside the network perimeter; in practice, mTLS between every pair) use the mesh.
4. Configuration stays per datacenter too. Consul's key/value store is not replicated across datacenters by default; replicate deliberately (a tool that copies a prefix from a primary datacenter) and treat the primary as the only writer.

Trade-offs and pitfalls

• Cross-DC failover can cause a cascade. If DC1 fails over its 10,000 requests per second onto DC2, DC2 needs that headroom. Cap failover traffic or pre-provision for it.
• DNS cache ignorance is the classic outage: some runtimes pin an answer far longer than the TTL. Test every major language's behaviour, do not assume.
• A mesh doubles your upgrade surface (every sidecar plus the control plane), and a control-plane outage must leave proxies serving their last-known endpoints, not empty ones.
• What would change the recommendation: a single datacenter makes most of the federation machinery unnecessary; a hard requirement for mTLS everywhere makes the mesh the default rather than the exception; an all-Kubernetes estate can use Kubernetes' native discovery per cluster plus a multi-cluster layer instead of Consul.

Direct answer

For about 200 services and 1,000 instances, I would choose Consul: it already is a service registry (catalog, health checks, DNS interface, watch-style queries), whereas etcd and ZooKeeper (distributed key-value stores built for coordination tasks like leader election and locking, not purpose-built service registries) are general coordination stores on which you would have to build registration, health checking and a lookup API yourself. The design is: 3 or 5 Consul servers forming a Raft cluster that holds the catalog; a Consul client agent on every host that registers local instances and runs their health checks locally; and clients discovering endpoints either through DNS (zero code change) or through blocking queries in a shared client library that caches healthy endpoints and load-balances client-side.

Terms

• Service registry: a directory mapping a service name (orders) to its live instances (10.1.4.7:8080, ...).
• Client-side vs server-side discovery: in client-side discovery the caller looks up instances and picks one itself; in server-side discovery the caller hits a load balancer or proxy that does the lookup.
• Raft and quorum: the servers replicate every write using the Raft consensus protocol; a write commits once a majority (floor(n/2) + 1) has it. 3 servers survive 1 failure; 5 survive 2.
• Gossip: Consul agents also exchange membership information peer to peer, using a protocol based on SWIM. In plain terms: each agent periodically picks a random peer and pings it directly; if there is no reply, it asks a few other random peers to ping that same target on its behalf, in case the problem is just the direct path between prober and target; if those also fail, the agent marks the target suspect and mentions that suspicion in its own next round of pings to other random peers, who do the same, so the news spreads through the cluster the way a rumor spreads through a crowd (hence "infection-style") without any node pinging a central server. This detects dead nodes without every node pinging the servers.
• Blocking query: an HTTP request that includes the last index the client saw; the server holds it open until the data changes (or a timeout passes), then answers. It gives near-instant change notification with ordinary HTTP.

Components

flowchart LR
  subgraph Host[Each host]
    Svc[Service instance] -->|register on start| Agent[Consul client agent]
    Agent -->|run health check locally| Svc
  end
  Agent -->|state changes, anti-entropy sync| Servers[3 or 5 Consul servers, Raft]
  Caller[Calling service with client library] -->|blocking query: healthy instances of orders| Servers
  Caller -->|or DNS: orders.service.consul| Agent
  Caller -->|direct connection| Svc

1. Servers (3, spread across availability zones): hold the catalog and key-value store, replicate with Raft. At 1,000 instances, 3 servers are enough; move to 5 when you want to survive a zone loss plus one more failure, or a failure during maintenance.
2. Client agents (one per host): accept registrations from local instances, run their health checks, and sync to the servers. That sync is called anti-entropy: the agent periodically reconciles its local view of what should be registered against the servers' catalog, so a missed update from a blip or a restart is corrected on the next pass instead of drifting forever.
3. Client library / DNS: the lookup path for callers.
4. Deploy integration: the deploy system or the instance's startup script registers, and graceful shutdown deregisters before the process stops.

APIs

Registration (instance to local agent):

PUT /v1/agent/service/register
{ "Name": "orders", "ID": "orders-7f3a", "Address": "10.1.4.7", "Port": 8080,
  "Tags": ["v2", "zone-a"], "Meta": {"version": "2.14.0"},
  "Check": { "HTTP": "http://10.1.4.7:8080/healthz", "Interval": "10s", "Timeout": "2s",
             "DeregisterCriticalServiceAfter": "5m" } }
PUT /v1/agent/service/deregister/orders-7f3a      (on graceful shutdown)

Reading the Check block: Consul calls the HTTP URL every Interval, allows Timeout for a reply, and if it stays failing for DeregisterCriticalServiceAfter, removes the registration entirely rather than just marking it unhealthy forever.

Discovery (caller):

GET /v1/health/service/orders?passing=true&index=<last_index>&wait=60s

dig @127.0.0.1 -p 8600 orders.service.consul SRV

The first returns only instances whose checks pass, and blocks until that list changes. The second uses Consul's DNS interface (port 8600 by default), which returns healthy instances and works for any language that can resolve a hostname.

Health-check model: active pings vs TTL vs passive

Choice: active HTTP checks as the default, TTL checks for workers without a port, and passive outlier ejection in the client library as a second layer, because it catches the gap between "health check passes" and "real requests fail". Checks run on the local agent, so they do not travel across the network, and Consul syncs status changes to the servers rather than every check result, which keeps server load independent of the check interval.

DeregisterCriticalServiceAfter removes instances that have been failing for 5 minutes, so crashed instances that never deregistered do not accumulate.

How clients discover endpoints

• Default: DNS. Any service can call orders.service.consul. Keep the DNS TTL short (a few seconds) so changes propagate, and remember that DNS gives addresses but not ports unless the client asks for SRV records (DNS service records, which carry a port and a hostname).
• Preferred for services owned by the platform: client library using blocking queries, holding the healthy list in memory, balancing across it (round robin, or least-requests: send the next call to whichever healthy instance currently has the fewest outstanding calls), with passive ejection and a stale-on-error fallback: if Consul is unreachable, keep using the last known list rather than failing every request.

Worked example: is this load a problem?

• Watches: 1,000 instances × about 5 dependencies each = 5,000 blocking queries held open. Blocking queries are designed for this; each is an idle held connection until something changes.
• A rolling deploy of a 20-instance orders service produces about 40 catalog changes (each old instance deregistered, each new one registered). If 50 caller instances watch orders, each change wakes 50 watchers: 40 × 50 = 2,000 responses. At roughly 20 KB per response (20 instances with tags and metadata), that is 2,000 × 20 KB = 40,000 KB, about 40 MB spread across the deploy window. Trivial for three servers.
• Registration writes: a few per second at most, even during a busy deploy hour. Raft write throughput is not the constraint at this size; operational simplicity is.

Justifying Consul operationally (vs etcd and ZooKeeper)

What would change my choice: if everything runs on Kubernetes, I would not add Consul at all: Kubernetes Services (a stable name for a group of pods), EndpointSlices (the live, auto-updated list of those pods' actual IPs) and cluster DNS already are a registry, with a readiness probe (Kubernetes' own periodic "can this pod take traffic yet" check) standing in for the health check. If the platform already runs etcd for other coordination and the team is Go-heavy, etcd with leases is reasonable. ZooKeeper only makes sense if it is already operated for Kafka-era or Hadoop-era systems (older big-data tools such as Apache Kafka and Hadoop that were built assuming ZooKeeper was available) and the team knows it well.

Pitfalls

• Health check says healthy, dependency is dead. Keep /healthz shallow (the process can serve) and rely on passive ejection for deep failures; a deep check that pings the database can take every instance out at once when the database blips.
• No graceful deregistration. Instances killed without deregistering keep receiving traffic until their check fails. Deregister, wait for callers to notice, then stop.
• Treating the registry as always available. Callers must cache and serve stale on error, or a registry outage becomes a whole-platform outage.

Direct answer

Split it into two systems with opposite consistency needs. Service discovery is region-local and eventually consistent: each of the 5 regions runs its own registry, clients discover only same-region endpoints, and no request ever waits on another region. Configuration has one global, strongly consistent write path that assigns every change a version and an audit record, then replicates asynchronously to a read-only config store in each region; each region points at a version, so a rollback moves one region's pointer back without touching the others. Users reach a region through geo-DNS with a short TTL (time-to-live) plus health checks, and every region is provisioned to absorb a failed neighbour's share.

Requirements and numbers I am designing to

• 1M requests per second (RPS) globally over 5 regions: about 200,000 RPS per region in steady state.
• Regional failover: if one region dies, its traffic spreads over the other 4, so each must handle 1,000,000 / 4 = 250,000 RPS. Each region is therefore provisioned at 25% above its normal load (250,000 / 200,000 = 1.25).
• Assume about 4,000 service instances per region (an illustrative sizing, not a measurement). Discovery load is driven by instance churn and watches, not by the 1M RPS: clients cache endpoint lists, so a request never calls the registry.
• Config writes are rare (tens per hour); config reads happen at process start and on every change, never per request.

Terms used below

• Registry: the database of "which healthy instances serve service X", e.g. Consul or the Kubernetes API server.
• Raft: a consensus protocol; a small cluster (3 or 5 nodes) elects a leader and commits a write only when a majority (the quorum) has stored it. It gives linearizable writes: once a write is acknowledged, every later read sees it.
• Strong vs eventual consistency: strong means every reader sees the latest committed value; eventual means readers may lag briefly but converge.
• Geo-DNS: an authoritative DNS service that answers the same name with different addresses depending on where the client is, and stops returning a region that fails health checks.

Architecture

flowchart TB
  U[Users] --> G[Geo-DNS with health checks]
  G --> R1[Region A edge LB]
  G --> R2[Region B edge LB]
  subgraph CP[Global config control plane]
    W[Config API, Git review] --> S[(Raft store: versions and audit log)]
  end
  S -->|async replication| C1[(Region A config cache)]
  S -->|async replication| C2[(Region B config cache)]
  R1 --> A1[Services in A]
  A1 --> D1[(Region A registry)]
  A1 --> C1
  R2 --> B1[Services in B]
  B1 --> D2[(Region B registry)]
  B1 --> C2

Two regions are drawn for readability; the other three are identical. Each region's edge LB (edge load balancer: the boundary layer that terminates incoming user connections and forwards them to services inside that region) sits behind geo-DNS and in front of the region's own services.

1. Region-local discovery

• Each region runs its own registry cluster (a 5-node Raft cluster across that region's availability zones, meaning physically separate data centers within the region, each with independent power and networking, so the cluster survives one of them failing, or the Kubernetes control plane per cluster). There is no global registry: a cross-region consensus round trip (tens to over a hundred milliseconds of network delay per commit) has no place in the discovery path, and a trans-oceanic partition must not break discovery inside a region.
• Health does not go through Raft. If 4,000 instances each wrote a heartbeat through consensus every 10 seconds, that is 4,000 / 10 = 400 consensus writes per second per region for nothing but liveness. Instead, a local agent or the kubelet (the per-node Kubernetes agent that already runs on every node and reports pod health) probes each instance and only state changes (healthy to unhealthy) become registry writes.
• Clients or sidecar proxies (a proxy process deployed next to each service instance, as in a service mesh) hold a watch on the registry and keep an in-memory endpoint list, so discovery reads cost nothing per request and survive a registry outage using the last known list.
• Endpoint entries carry version tags (orders v42), so a rolling upgrade can route by version, keep old and new side by side, and drain old instances cleanly.

2. Automatic regional failover

• User to region: geo-DNS answers with the nearest healthy region. Health checks probe a deep endpoint in each region (one that exercises the region's real dependencies) every 10 seconds and fail it after 3 misses. DNS TTL is 30 seconds, so worst-case steering time is about 30 s detection + 30 s TTL = 60 s, plus clients that hold connections open (cap connection lifetime so they re-resolve). An anycast global load balancer (one IP address announced from every region at once, so network routing itself, not DNS or client caching, sends each user to the nearest healthy region and can redirect within seconds of a region failing) can replace geo-DNS and fail over in seconds without depending on client DNS caches; I would use one where available and keep geo-DNS as the fallback.
• Service to service: stays in-region. If a single service is down in region A but the region is otherwise up, the local proxy can fail over to that service in the nearest region, but only when the registry's failover policy allows it, because it adds cross-region latency and egress cost (what cloud providers charge for data leaving a region) and can overload the neighbour.
• Capacity: the 25% headroom above is non-negotiable; without it a regional failover becomes a multi-region outage as the surviving regions overload one after another.

3. Versioned global config with audit history

• Config lives in a single global control-plane store: a 5-node Raft cluster with one node per region. Writes need 3 of 5 regions, so a write commits even with 2 regions unreachable, and each commit costs one cross-region round trip (the leader sends the write to the other 4 regions in parallel and only has to wait for the fastest 2 replies, since itself plus those 2 is already a majority of 3; because the sends and replies happen in parallel, it is one round trip, not four sequential ones), acceptable for tens of writes per hour.
• Every write is: reviewed change in Git, then applied through the config API as (key, value, version, author, change_id, reason) with a monotonically increasing version, appended to an append-only audit log in the same transaction. Nothing edits config directly in a region.
• Each region runs a read-only config replica fed by an asynchronous stream from the control plane. Services read only from their region's replica (via a local agent that watches it and writes a file or pushes to the process). If the control plane is unreachable, regions keep serving last known good config: config staleness is acceptable, config unavailability is not.
• Replicas publish their applied version; an alert fires if any region's version lags the global head by more than a few minutes.

4. Region-scoped rollouts and rollbacks

• The control plane stores immutable versions plus, per region, a pointer: region-a -> v118, region-b -> v117. A rollout advances pointers region by region with a bake time between (a deliberate pause after each region's change, watching its metrics before touching the next one), starting with the lowest-traffic region.
• Rollback of region B is a single write: region-b -> v117. The versions themselves never change, so the rollback is exact, audited like any other change, and does not affect regions A, C, D, E.
• Every config read reports the version it served, so "which version is region B running" is answered from telemetry, not from what the control plane intended.

Strong versus eventual consistency: where each belongs

The rule behind the table: pay for strong consistency only where a stale or conflicting value is a correctness bug, and never on the request path.

Trade-offs and pitfalls

• A single global registry (one Raft cluster spanning all 5 regions for discovery) is the common wrong design: every instance change pays a cross-region commit and a partition can stall discovery everywhere.
• Async config replication means a brief window where regions run different versions. That is intended (it is how region-by-region rollout works), but configs must be backward compatible across adjacent versions.
• Low DNS TTLs are not failover by themselves: pooled connections and runtime DNS caches ignore them, and the TTL only starts after health checks detect the failure.
• What would change the design: if config must be read-your-writes consistent across regions (read-your-writes consistent: a client that just wrote a value is guaranteed to see that same value on its very next read from any region, never a stale one; for example a security revocation), send it through the control plane with a synchronous "applied in all regions" acknowledgement instead of the async path; if regions are fully independent tenants, drop the global Raft and run five independent config stores synced from Git.

Direct answer

Service discovery is how one service finds the current network addresses of another when instances come and go all the time (autoscaling, deploys, crashes). A service registry keeps the list of live instances. In client-side discovery, the calling service asks the registry for the list and picks an instance itself, so the client does the load balancing. In server-side (proxy) discovery, the caller sends the request to one stable address (a load balancer or proxy), and that component looks up the instances and picks one. My default is server-side through the platform (a Kubernetes Service or a load balancer) because it keeps clients simple and language-independent; I switch to client-side when the extra hop or the lack of per-request control actually costs something.

Why it is needed

In a static world you would hard-code payments = 10.0.1.5. In a microservice fleet, instances get new IPs on every deploy and the count changes with load, so addresses must be looked up at runtime, and the lookup must only return instances that are healthy (able to serve).

The two approaches

Client-side discovery

1. Each payments instance registers itself (or is registered by the platform) in the registry.
2. The checkout service queries the registry (or a local cache of it) for payments and gets a list, such as [10.0.1.5, 10.0.1.9, 10.0.2.4].
3. checkout's own library picks one (round robin: cycle through the list in order; or least outstanding requests: send to whichever instance currently has the fewest in-flight calls) and connects directly.

Server-side (proxy) discovery

1. Instances register the same way.
2. checkout calls one stable name, such as payments.default.svc.cluster.local or a load-balancer address.
3. The load balancer or proxy (which watches the registry) chooses an instance and forwards the request.

Who does what

On health checks: the registry usually learns health by probing instances (an HTTP call to /health every few seconds) or by requiring heartbeats. The client or proxy adds a second, faster signal: passive health checking, which stops sending to an instance that just returned several errors in a row, without waiting for the next probe.

Real-world tools

• DNS: a service name resolves to several A records (IP addresses) and the client picks one. That is a simple form of client-side discovery, but with weak health handling and caching delays (clients keep an answer until its TTL, time-to-live, expires, and some ignore TTLs).
• Consul: supports both. Clients can query its HTTP API or its DNS interface (payments.service.consul, returning only healthy instances) and pick an instance themselves: client-side. Or traffic goes through Consul's service mesh, where a sidecar proxy (a helper process running next to each service) does the choosing: server-side, but on the same host.
• Kubernetes Service: server-side. A normal Service gets one stable virtual IP (the ClusterIP); kube-proxy (the Kubernetes component that runs on every node and keeps its routing rules in sync with the cluster) programs rules, either iptables (the Linux kernel's built-in packet-filtering and NAT table) or IPVS (the Linux kernel's built-in load balancer), that spread connections across the pods (a pod is Kubernetes' smallest deployable unit, one or more containers scheduled together) that are passing their readiness probe (a periodic check, such as an HTTP call, that decides whether a pod is currently able to serve traffic; failing it removes the pod from the Service without killing it). A headless Service (no ClusterIP) instead returns the pod IPs through DNS, which is client-side, and is what gRPC (a high-performance RPC framework built on HTTP/2, which multiplexes many concurrent requests over a single long-lived TCP connection instead of opening a new one per call) clients often use to balance per request.
• Cloud load balancers (for example, an AWS Application Load Balancer with a target group): server-side, with the cloud provider registering instances.
• Client-side libraries: gRPC's built-in load-balancing policies, Spring Cloud LoadBalancer (which replaced the Netflix Ribbon library that is no longer developed).

Worked example

checkout runs 200 instances and calls payments, which runs 50.

• Client-side: every checkout instance may keep connections to every payments instance: 200 × 50 = 10,000 connections. When payments instance 17 starts timing out, each checkout instance notices within its own next few requests and stops using it, without waiting for the registry. But the balancing code has to exist, and be kept correct, in every language checkout-like services are written in.
• Server-side with a Kubernetes Service: checkout knows one address. Connections are spread by kube-proxy, which balances per connection, not per request: it picks a pod once, when a TCP connection is first opened, and every request sent over that connection afterward goes to that same pod. Since gRPC keeps one HTTP/2 connection open and multiplexes many requests over it (see above), a checkout instance that opens one long-lived HTTP/2 connection sends all of its requests, forever, to whichever single pod answered first, which is a classic cause of uneven load for gRPC. Fixes: a headless Service plus client-side balancing, or a proxy that balances per request.

When to choose each

• Choose server-side when you have many languages, want one place to change routing, or run on a platform that already provides it (Kubernetes, cloud load balancers). This is the right default for most teams.
• Choose client-side when the extra hop matters (very latency-sensitive internal calls), when you need per-request balancing of long-lived connections (gRPC), or when you want each caller to react to failures instantly.
• A service mesh blends them: the logic lives in a proxy (server-side style, one implementation for all languages), but the proxy sits next to each client (client-side style, no central hop). It costs you operating the mesh.

Pitfalls

• Registering an instance as soon as the process starts, before it can serve (use readiness, not liveness, to decide when it receives traffic: a liveness probe only answers "is this process stuck and needs restarting", it says nothing about whether the process is ready to handle requests right now).
• Clients that cache the instance list with no refresh, so they keep calling instances that were removed.
• Treating the registry as the only health signal: a registry probe every 10 s means up to 10 s of requests to a dead instance unless the caller also reacts to errors.

Direct answer

The key decision is to evaluate flags inside the application process, not over the network. Each service embeds an SDK (software development kit, the client library) that holds the full rule set in memory and evaluates a flag with a hash and a few comparisons, so a million evaluations per second is a million local function calls with no network hop and no central bottleneck. A central control plane stores flags as versioned, immutable documents with an append-only change log, and pushes each new version to regional relays that stream it to every SDK within seconds. Percentage rollouts use a deterministic hash of flag and user, so a user's answer never flickers. Rollback is "publish the previous version", and the kill switch is a rule evaluated before all others.

Terms first

• Feature flag: a named switch whose value is decided at runtime per request, so behaviour changes without a deploy.
• Targeting: rules like "users in segment beta-testers get on", "country = DE gets off".
• Segment: a reusable named group of users, defined either by a rule (plan = enterprise) or an explicit list of IDs.
• Deterministic percentage rollout: "25% of users" chosen by hashing, so the same user always lands in the same group without storing anything.
• Kill switch: a way to force a flag (or all flags of a feature) to its safe value instantly.
• SSE (server-sent events): a long-lived HTTP response over which a server pushes messages to a client; simpler than WebSockets for one-way updates.
• Control plane: the central service that owns the source of truth for every flag and pushes changes out; it never evaluates a flag for live traffic itself, it only distributes the rules.
• Relay: a regional fleet of servers between the control plane and the SDKs. It holds a cached copy of the current rules and streams changes to every SDK connected to it, so the control plane itself is never in the request-serving path.
• Exposure event: a small record ("user U-482 saw flag new-checkout as on") sampled and logged when a flag is evaluated, used afterwards to measure the flag's effect. It plays no part in serving the flag.

Requirements I am designing to

• 1,000,000 evaluations per second in total, low latency (target: microseconds per evaluation, so it can sit on every request path).
• Several regions; a region losing contact with the control plane must keep evaluating correctly on last-known rules.
• Flag change visible everywhere in seconds; rollback in seconds; every change attributable and reversible.

Architecture

flowchart LR
  UI[Admin UI and API] --> CP[Control plane: validation, approvals]
  CP --> DB[(Primary DB: flag versions and change log)]
  CP --> BUS[Change stream]
  BUS --> R1[Region A relay fleet]
  BUS --> R2[Region B relay fleet]
  CP --> SNAP[Versioned snapshots in object storage]
  R1 -->|SSE stream| S1[Service SDKs, in-process eval]
  R2 -->|SSE stream| S2[Service SDKs, in-process eval]
  SNAP -.->|bootstrap and fallback| S1
  SNAP -.->|bootstrap and fallback| S2
  S1 --> EXP[Sampled exposure events]

Runtime evaluation model

Each flag is a small document: variations (on/off, or variants like A/B), an ordered rule list, a default, and a version number. The SDK evaluates in a fixed order:

1. Kill switch / flag off → return the off variation. Checked first so nothing else can override it.
2. Individual targets (explicit user IDs).
3. Segment rules in order ("in segment beta-testers" → on).
4. Percentage rollout on the fallthrough (reached only if none of the rules above matched): hash bucket below the threshold → on.
5. Default.

Traced example: user user-000004 requests the new-checkout flag at a live 25% rollout. Step 1, the kill switch: not tripped. Step 2, individual targets: user-000004 is not on the explicit list. Step 3, segments: not a member of any targeted segment. Step 4, percentage rollout: bucket("new-checkout", "s1", "user-000004") comes out to 1,531, and 1,531 is below the 25% threshold of 2,500, so the flag resolves to on right here. Step 5, the default, is never reached for this user.

A global kill switch is a special entry in the rule set ("emergency: all flags tagged checkout return their off variation") that the SDK checks before any flag, so one change disables a whole feature area.

Browsers and mobile apps should not receive the full rule set (it exposes targeting logic and user lists), so they call a thin server-side evaluation endpoint on the relay that returns only their evaluated values.

Deterministic percentage rollouts

Hash flag_key : salt : user_id into 10,000 buckets (the salt is an arbitrary fixed string with no meaning of its own beyond changing the hash's output); the user is in a 25% rollout when their bucket is below 2,500. Including the flag key means different flags pick independent user groups; the salt lets you deliberately reshuffle a flag.

import hashlib

def bucket(flag_key, salt, user_id):
    """Map (flag, user) to a stable bucket in [0, 10000)."""
    h = hashlib.sha256(f"{flag_key}:{salt}:{user_id}".encode()).digest()
    return int.from_bytes(h[:8], "big") % 10_000

def in_rollout(flag_key, salt, user_id, percent):
    return bucket(flag_key, salt, user_id) < percent * 100   # 25% -> buckets 0..2499

users = [f"user-{i}" for i in range(100_000)]
at10 = {u for u in users if in_rollout("new-checkout", "s1", u, 10)}
at25 = {u for u in users if in_rollout("new-checkout", "s1", u, 25)}
other = {u for u in users if in_rollout("new-search", "s1", u, 10)}
print("10% cohort:", len(at10), " 25% cohort:", len(at25))
print("everyone in 10% still in 25%:", at10 <= at25)
print("overlap with a different flag's 10%:", len(at10 & other))

Output (Python 3, run as shown):

10% cohort: 9997  25% cohort: 25083
everyone in 10% still in 25%: True
overlap with a different flag's 10%: 986

Raising a rollout from 10% to 25% only adds users (nobody who had the feature loses it), and a second flag's 10% overlaps this one's at about 10% of 10% (986 of 100,000), which is what independent selection looks like. Every SDK in every language must implement the identical hash, so the algorithm is part of the service's contract and is covered by cross-language test vectors (the same fixed set of (flag, salt, user) inputs and their expected bucket numbers, run against every language's SDK, so a Python evaluation and a Java evaluation of the same user never disagree).

How a change propagates

1. Admin saves a change. The control plane validates it (schema, no dangling segment references, approval if the flag is tagged production-critical), writes flag version N+1 and a change-log entry in one transaction, then publishes to the change stream.
2. It also writes a full versioned snapshot to object storage (fronted by a CDN, content delivery network), used for SDK bootstrap and as a fallback.
3. Regional relays consume the stream and push the delta (only what changed since the SDK's last known version, not the whole rule set) to every connected SDK over SSE. Each message carries the version, and the SDK ignores anything older than what it has, so reordering or replays are harmless.
4. If the stream drops, the SDK keeps evaluating on its current rules, reconnects with backoff, and falls back to polling the snapshot. It never goes blank: an SDK that has not loaded rules yet returns the code-level default for every flag, which must be the safe value.

Worked example: capacity arithmetic

Inputs (assumptions to state in the interview): 20,000 application instances running the SDK, 2,000 flags, about 2 KiB per flag document, exposure events sampled at 1%.

evals_per_s = 1_000_000
sdk_instances = 20_000          # app processes running the evaluator (assumption)
flags, bytes_per_flag = 2_000, 2_048
ruleset = flags * bytes_per_flag
print(f"per-instance eval rate: {evals_per_s / sdk_instances:.0f}/s")
print(f"full ruleset: {ruleset / 2**20:.1f} MiB held in memory per instance")
print(f"one flag change fanned out to all instances: {sdk_instances * bytes_per_flag / 2**20:.1f} MiB")
print(f"full resync of every instance (e.g. after a relay restart): {sdk_instances * ruleset / 2**30:.1f} GiB")
exposure_sample = 0.01
print(f"exposure events at 1% sampling: {evals_per_s * exposure_sample:,.0f}/s")
changed_since_reconnect = 5      # flags changed while an SDK was briefly disconnected (assumption)
delta_bytes = changed_since_reconnect * bytes_per_flag
print(f"delta resume after missing {changed_since_reconnect} changes: {delta_bytes / 2**10:.1f} KiB per instance, "
      f"vs the {ruleset / 2**20:.1f} MiB full ruleset ({ruleset / delta_bytes:.0f}x less)")
print(f"same delta, fleet-wide: {sdk_instances * delta_bytes / 2**20:.1f} MiB vs the {sdk_instances * ruleset / 2**30:.1f} GiB full resync")

Output (Python 3, run as shown):

per-instance eval rate: 50/s
full ruleset: 3.9 MiB held in memory per instance
one flag change fanned out to all instances: 39.1 MiB
full resync of every instance (e.g. after a relay restart): 76.3 GiB
exposure events at 1% sampling: 10,000/s
delta resume after missing 5 changes: 10.0 KiB per instance, vs the 3.9 MiB full ruleset (400x less)
same delta, fleet-wide: 195.3 MiB vs the 76.3 GiB full resync

What this tells you: evaluation load is trivial (50 per second per instance), memory is small (about 4 MiB), and a normal change is cheap (about 39 MiB of fan-out). The dangerous number is the 76.3 GiB full resync: if every SDK reconnects at once and downloads everything from the relays, that is a thundering herd (a mass of clients all retrying the same operation at the same moment, turning ordinary load into a spike large enough to overwhelm whatever they are hitting). Mitigations: SDKs reconnect with jittered backoff (waiting a random, growing interval before retrying, so reconnects do not all land on the same instant), resume with "I have version N, send deltas" (5 changed flags costs 10.0 KiB per instance against the 3.9 MiB full ruleset, about 400x less, and 195.3 MiB fleet-wide against the 76.3 GiB full resync), and bootstrap full snapshots from the CDN rather than the relays.

Storage and caching

• Source of truth: one relational database in a primary region (flag documents keyed by (flag_key, version), never updated in place). Writes are rare (humans making changes), so single-region writes are fine; reads never touch it on the evaluation path.
• Regional caches: relays hold the current rule set in memory, rebuilt from the latest snapshot on start. The SDK's in-memory copy is the cache that serves all traffic.
• Multi-region: if the primary region is down, evaluation continues everywhere on last-known rules; only editing is unavailable. That is the right trade: editing flags during a regional outage is rare and can use a documented break-glass path in a standby region.

Audit trail and fast rollback

• The change log is append-only: actor, time, flag, old and new version, diff, reason or ticket link, approvals. Nobody, including admins, can edit or delete an entry; it is streamed to the company's audit store as well.
• Rollback = publish the previous version as N+2. Because versions are immutable, "restore version 41" is one click and propagates at the same speed as any change (seconds), and the rollback itself appears in the audit log.
• Kill switch for emergencies needs no approval and flips the flag to off; everything else can require review.

Trade-offs and pitfalls

• Local evaluation vs central evaluation. Central evaluation (every check is an RPC, remote procedure call) makes the flag service a dependency of every request: 1M requests per second of load, a network hop on every call, and an outage takes down every product. Local evaluation costs rule distribution and a multi-language SDK effort. At this scale local wins; with only a few services and low volume, a central endpoint would be simpler.
• Large list segments (millions of user IDs) do not fit in every SDK. Keep those as server-side segment lookups or push them only to services that need them.
• Stale flags accumulate as debt: track last-evaluated time from exposure events and nag owners of flags that have been 100% on for 30 days.
• Consistency across a request: two services may briefly see different versions during a push. For flows where that matters, evaluate once at the edge and pass the result downstream in a header.