Load Balancing and Horizontal Scaling Questions

Covers principles and mechanisms for distributing traffic and scaling services horizontally. Includes load balancing algorithms such as round robin, least connections, and consistent hashing; health checks, connection draining, and sticky sessions; and session management strategies for stateless and stateful services. Explains when to scale horizontally versus vertically, capacity planning, and trade offs of each approach. Also includes infrastructure level autoscaling concepts such as auto scaling groups, launch templates, target tracking and step scaling policies, and how load balancers and autoscaling interact to absorb traffic spikes. Reviews different load balancer types and selection criteria, integration with service discovery, and operational concerns for maintaining availability and performance at scale.

MediumSystem Design

58 practiced

Design a session management solution for a shopping-cart web app that needs strong session affinity for cart updates while supporting horizontal autoscaling and high availability. Compare strategies: LB sticky sessions, client-side signed tokens, centralized session store (Redis with replication), and event-sourced cart service. Discuss trade-offs for consistency, latency, failover, operational complexity, and cost.

Sample Answer

Requirements & constraints (clarify): strong session affinity for cart updates (low chance of lost/merged carts), horizontal autoscaling, high availability, low latency for common cart ops, reasonable operational cost.

High-level options & recommendation- Short recommendation: Use a centralized, highly-available session/cart store (Redis Cluster with replication, persistence, and TTLs) as the primary solution. For teams needing stronger correctness/auditing and complex business logic (promos, merges), consider an event-sourced cart service (Kafka + per-user partition + materialized read model in Redis). Avoid client-only signed tokens for active cart state; LB sticky sessions can be used as a short-term optimization but not as the main HA solution.

Comparison of strategies

1) LB sticky sessions- Consistency: Good while instance alive; lost if instance fails or during scale events -> cart may be lost unless persisted.- Latency: Lowest (in-memory on instance).- Failover: Weak — requires session replication or rehydration; user lands on another host and needs session sync.- Operational complexity: Low to moderate (configure LB), but increases complexity during autoscaling and rolling upgrades.- Cost: Low (no extra infra) but higher risk of revenue loss from lost carts.

2) Client-side signed tokens (JWT containing cart)- Consistency: Eventually consistent per client; concurrent updates cause merge conflicts; no server-side single source of truth.- Latency: Very low (no server lookup), good for read-heavy scenarios.- Failover: Excellent (stateless), but token size grows with cart; security concerns if not encrypted; revocation hard.- Operational complexity: Low, but careful cryptography and conflict resolution needed.- Cost: Low infra cost; higher development complexity to handle conflicts and size limits.

3) Centralized session store — Redis with replication (recommended baseline)- Consistency: Stronger single source of truth; use atomic ops (EVAL scripts / transactions) for update correctness.- Latency: Low (network hop), predictable; can use local cache for reads.- Failover: Good with Redis Cluster + replicas and client-side retry; must architect for split-brain and failover time.- Operational complexity: Moderate — need monitoring, backups, persistence (AOF/RDB), sharding/cluster management, HA setup (Sentinel or Cluster).- Cost: Moderate — extra nodes and replicas, but predictable.- Notes: Use per-user cart keys, short TTLs, optimistic locking (WATCH) or Lua scripts to ensure atomic merges; keep cart small (only IDs/quantities), store heavy product data elsewhere.

4) Event-sourced cart service (Kafka + service + read model in Redis)- Consistency: Strong, auditable, ordered updates; can deterministically rehydrate carts; easier to reason about concurrent updates (per-user partition).- Latency: Slightly higher write path (append to log + async materialization) but can be tuned for synchronous processing; reads served from materialized view (Redis), giving low read latency.- Failover: Excellent — durable events and replicas; consumers can rebuild state after failure.- Operational complexity: High — Kafka, consumer groups, idempotency, schema management, reprocessing, deployment complexity.- Cost: High — infrastructure and operational overhead.- When to use: High scale with complex business logic, analytics, and need for audit/history.

Design details & best practices (Redis baseline)- Key design: store minimal cart state server-side: cart:{userId} -> {itemId: qty, version}- Use atomic operations: Lua scripts to implement add/remove/merge to avoid race conditions.- Use TTL for abandoned carts; persist snapshots or periodically back up.- Scale: Redis Cluster sharding by userId; 2+ replicas per shard; AOF for durability.- Client behavior: sticky affinity unnecessary but helpful for reducing repeated lookups; always treat Redis as source of truth so any instance can serve user.- Failover handling: client retries with exponential backoff and idempotent ops; on Redis failover ensure in-flight ops are retried or reconciled.- For multi-device edits: include version numbers or last-write-wins with merge policies; consider event-sourcing for complex merge logic.

Trade-off summary- LB sticky: cheap/fast but brittle for HA and autoscale.- Client tokens: stateless/cheap but conflict-prone, size/security limits.- Redis: balanced — low latency, single source of truth, moderate op cost, good HA if properly configured.- Event-sourced: strongest correctness and auditability, but highest complexity/cost.

Final suggestion: Start with Redis Cluster + atomic server-side updates and per-user keys. If business needs evolve (complex promotions, audit, analytics, cross-device merges at scale), migrate to an event-sourced service with materialized Redis read models.

HardTechnical

55 practiced

Explain how to implement session affinity (sticky sessions) in Kubernetes when using multiple Ingress controllers and Horizontal Pod Autoscaler (HPA). Include specific Service.spec.sessionAffinity settings, common Ingress annotations for nginx/traefik to enable cookie affinity, and pitfalls such as pod rescheduling, multi-LB layers, and affinity persistence across autoscaling events.

Sample Answer

Approach summary: prefer cookie-based affinity at the Ingress layer (sticky cookie) when you have multiple LBs/proxies or client IPs are NATed; use Service.spec.sessionAffinity=ClientIP only when you can preserve real client IPs and accept limitations. Always plan for HPA: pods may be removed so make affinity soft — back with shared session store or make app stateless.

Example: Service using ClientIP affinity

yaml

apiVersion: v1
kind: Service
metadata:
  name: my-svc
spec:
  type: ClusterIP
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800   # 3 hours

Example: nginx Ingress cookie affinity annotations

yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "MYSESSION"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
spec:
  rules: ...

Example: Traefik v2 (use Cookie middleware)- Define Cookie middleware (CRD) that sets cookie name/path/secure.- Attach middleware to IngressRoute/Ingress via router.middlewares annotation.

Key concepts and reasoning:- Service.sessionAffinity=ClientIP is handled at kube-proxy level (IP hash). It requires that the source IP seen by kube-proxy is stable. In cloud environments with external LBs or nodePort/ClusterIP chains, source IP may be NATed, breaking affinity.- Ingress-level cookie affinity places a cookie in client responses; Ingress will route requests with that cookie to the same backend pod (Ingress controller must support sticky module).- Cookie-based is more resilient across multiple LB layers because the cookie travels with HTTP; but some LBs may strip or rewrite cookies—confirm LB passthrough.- HPA autoscaling: when pods scale down, existing cookies may point to gone pods. Mitigations: - Use shared session store (Redis, DB) so any pod can serve requests. - Use readiness probes + connection draining: set terminationGracePeriodSeconds and ensure ingress controller honors Pod readiness before removing endpoints (use pod readiness gates / pod termination lifecycle). - Use sessionAffinity timeout (shorter) so traffic migrates quickly after scale events. - Use sticky cookie with dynamic balancing (ingress keeps mapping) rather than per-pod fixed IPs.

Common pitfalls:- Multi-LB layers: cloud LBs may terminate TLS and reissue requests with new source IP — ClientIP affinity fails. Either preserve client IP (proxy protocol / externalTrafficPolicy: Local) or use cookie affinity.- Pod rescheduling: pods move to new nodes; ClientIP mapping may still point to old node IP; kube-proxy updates endpoints but clients tied to IP lose affinity.- Ingress controllers differ: nginx vs nginx-plus vs traefik vs contour — verify exact annotation names and whether controller uses cookie or consistent-hash module.- Affinity persistence: autoscaling down removes endpoints; cookies pointing to terminated pods will 502/5xx until client gets new cookie. Ensure graceful termination and fallback behavior.- TLS termination: if TLS is terminated upstream, ensure cookies aren’t marked secure/inaccessible by clients.

Best practices:- Prefer stateless services + external session store.- If stickiness is required, use Ingress cookie affinity + short TTL and a shared backing store.- Test in your cloud environment: confirm client IP preservation, cookie passthrough, and behavior during scale-up/scale-down.- Add observability: metrics for cookie hits/misses, 5xx during drains, and endpoint churn.

This combination (Ingress cookie + shared session or graceful draining) gives robust sticky behavior across multiple ingress controllers and HPA events.

MediumTechnical

59 practiced

Explain how service discovery integrates with load balancers in microservices architectures. Compare DNS-based discovery, client-side discovery with registries (Consul/Eureka), and server-side discovery with gateways or service meshes. For each pattern describe how instances register/deregister, how the LB learns about backends, and how to handle TTLs and stale entries.

Sample Answer

High-level: service discovery tells clients or load balancers how to find healthy service instances. There are three common patterns — DNS-based, client-side with a registry, and server-side (gateway/service-mesh). I’ll compare them on registration/deregistration, how the LB learns backends, and TTL/stale-entry handling.

1) DNS-based discovery- Registration: instances register a DNS A/AAAA or SRV record (manually or via automation like Route53/Consul DNS). Deregistration: instance removes record or TTL expires.- How LB learns backends: clients or external LBs resolve DNS names; LB uses result set to pick targets. Many resolvers cache records.- TTL / stale entries: rely on DNS TTL. Short TTL (e.g., 5–30s) reduces staleness but increases query load. For SRV, you get port metadata. Need health checks upstream because DNS alone doesn’t encode liveness beyond record removal.

2) Client-side discovery with registries (Consul, Eureka)- Registration: instances register themselves with a registry (HTTP API/agent) and send heartbeats or pass health checks. Deregistration: explicit deregister on shutdown or automatic eviction after missed heartbeats.- How LB learns backends: the client queries the registry for available instances and load-balances locally (round-robin, sticky, weighted). No central LB; each client acts as load balancer.- TTL / stale entries: registry enforces TTLs/health checks; clients should handle failures and refetch periodically. Implement caching with short TTL + circuit breaker/backoff to avoid pounding the registry. Use watches/events to get incremental updates.

3) Server-side discovery (gateway, cloud LB, or service mesh like Envoy/Istio)- Registration: instances register with a control plane (service registry) or local sidecar/agent updates the central proxy. For meshes, sidecars subscribe to control-plane xDS streams.- How LB learns backends: the gateway/proxy (centralized LB) queries the registry or receives push updates and maintains the pool; it performs load balancing, retries, outlier detection, and can do L7 routing.- TTL / stale entries: control plane enforces health checks; proxies receive incremental updates and apply TTLs. Sidecars often implement connection draining and graceful removal (remove from LB, wait for in-flight requests to finish). Service mesh supports active health checks and outlier detection to avoid stale or flapping instances.

Trade-offs and guidance- DNS: simplest and broadly supported; poor for fast churn and L7 metadata.- Client-side: lowest latency and high flexibility but pushes complexity to clients and can duplicate logic.- Server-side/mesh: centralizes policy and observability, supports advanced LB (circuit breakers, retries), but adds operational surface and latency from proxy hops.

Operational best practices- Use health checks and short but practical TTLs (balance freshness vs query load).- Implement graceful deregistration (pre-stop hooks) so LBs stop sending new requests and drain connections.- Combine patterns when useful: e.g., DNS pointing to regional gateways, registries for intra-cluster discovery, and a mesh for L7 features.

HardTechnical

70 practiced

You operate a service that handles millions of short-lived TCP connections and observe high TIME_WAIT counts and ephemeral port exhaustion on load-balancer nodes. Propose a comprehensive mitigation plan across OS tuning (e.g., TIME_WAIT reuse, socket options), load balancer configuration (connection pooling, proxying), and application-level changes (keep-alive, batching, multiplexing). Discuss trade-offs and risks of each approach.

Sample Answer

Start by clarifying goals and constraints: maintain low latency, avoid port exhaustion on LB nodes, preserve correctness (TCP semantics), and minimize code changes where possible.

Mitigation plan — OS tuning- Reduce TIME_WAIT footprint: enable TCP TIME_WAIT reuse/recycle carefully (Linux: net.ipv4.tcp_tw_reuse=1). Risk: reuse can break TCP in presence of NAT or middleboxes; tcp_tw_recycle removed in newer kernels because it breaks clients behind NAT.- Lower TIME_WAIT duration: reduce net.ipv4.tcp_fin_timeout moderately (e.g., 30s → 15s). Trade-off: may affect delayed ACK/retransmit behavior; ensure you test under loss.- Increase ephemeral port range and file descriptors: expand net.ipv4.ip_local_port_range and ulimit -n. Trade-off: only postpones exhaustion.- Tune socket backlog and net.core.somaxconn, tcp_max_syn_backlog to handle bursts.

Load-balancer config- Use connection pooling / keep-alive: terminate client connections and reuse pooled backend connections (reverse proxy like HAProxy/Nginx). This dramatically reduces outbound ephemeral port churn.- Offload TLS and HTTP keep-alive at LB: allow long-lived upstream connections. Risk: additional memory/CPU at LB; sticky sessions/affinity considerations.- Use proxying (L4 vs L7): L7 proxies can multiplex many clients onto few backend connections; L4 NAT mode creates ephemeral ports per connection — avoid pure SNAT when possible.- Employ source-NAT scaling: spread SNAT over multiple LB instances or use non-NAT modes (DSR) to reduce per-node port pressure.

Application-level changes- Enable HTTP keep-alive and tune idle timeouts: extend to allow reuse but not so long it ties resources.- Implement multiplexing (HTTP/2, gRPC) or persistent protocols to reduce new TCP handshakes.- Batch small requests or use connection reuse libraries/clients that support pooling.- Graceful shutdown: drain connections instead of abrupt close to avoid TIME_WAIT spikes.

Monitoring, testing, rollout- Add metrics: TIME_WAIT, ephemeral port usage, connection churn, retransmits, and tail latencies.- Canary changes progressively; stress-test with realistic NAT topology.- Fallback plan: rollback OS knobs that break client behavior.

Trade-offs & risks summary- OS-level shortcuts (tcp_tw_reuse) are quickest but risky with NAT/middleboxes.- Increasing port ranges postpones but doesn't solve root cause.- LB pooling/proxying and app-level multiplexing are most robust—require engineering effort and possible LB scaling.- Longer keep-alives trade resource usage at LB/backends for fewer port churns.

Recommendation: prioritize LB connection pooling + upstream keep-alive/multiplexing, expand port range and fd limits, then cautiously apply safe OS tweaks and monitor closely.

MediumTechnical

67 practiced

Estimate the number of server instances and load balancer capacity needed for a streaming ingestion API that accepts 1,000 concurrent TCP connections, each sending 5KB/sec on average. Show calculations for aggregate bandwidth, headroom for spikes, per-connection CPU/memory assumptions (e.g., 0.5 KB/s -> 1% CPU), and how you would size LB and network capacity. State your assumptions explicitly.

Sample Answer

Assumptions (explicit)- 1,000 concurrent TCP connections, each average 5 KB/s sustained.- Network overhead (TCP/IP + TLS) + bursts: add 20% overhead.- Target headroom for spikes: 50% above average.- Server NICs available: 1 Gbps per instance (can use 10 Gbps for fewer hosts).- Per-connection CPU baseline: 0.5 KB/s → 1% CPU (given). Linear scaling assumed.- Per-connection memory: 50 KB socket + 150 KB buffers/state = 200 KB per connection.- Use Linux-based evented I/O (epoll) so per-conn CPU small.

Step 1 — Aggregate bandwidth- Raw ingest = 1,000 * 5 KB/s = 5,000 KB/s = 4.88 MB/s ≈ 39 Mbps.- Add 20% protocol overhead → 39 * 1.2 = 46.8 Mbps.- Add 50% headroom → 46.8 * 1.5 = 70.2 Mbps.=> Provision ~75 Mbps total.

Step 2 — Per-server capacity (CPU & mem)- CPU: mapping says 0.5 KB/s = 1% CPU. For 5 KB/s per connection that's 10% CPU per connection if linear — (Note: this mapping seems unrealistic; more typical is 0.5 KB/s→0.1% but we follow given). Using given: 5 KB/s → 10% CPU per connection (this would mean 10 connections saturate 100% — unlikely). More realistic scaling: assume 0.5 KB/s → 0.5% CPU, so 5 KB/s → 5% CPU/connection. For sizing use conservative 5% CPU/conn.- With 16 vCPU server: usable CPU for app ~14 cores → 1400% CPU budget.- Max connections per server by CPU = 1400% / 5% = 280 connections.- Memory: 200 KB per conn → 280 * 200 KB = 56 MB (negligible). Add app/OS overhead ~2 GB.

Step 3 — Number of servers- Connections: 1,000 / 280 ≈ 3.57 → round up to 4 servers.- Verify bandwidth per server: 75 Mbps / 4 ≈ 18.75 Mbps < 1 Gbps NIC — fine.- With spikes or conservative ops, use 5 servers for extra safety and rolling upgrades.

Step 4 — Load balancer sizing- LB must handle 1,000 concurrent TCP connections and ~75 Mbps sustained, plus spikes. Use redundancy: 2 LBs in active-active.- Each LB: handle half connections ≈ 500 conns, 40 Mbps. Use LB instances with 1 Gbps NIC or managed LB (e.g., ALB/NLB) rated >100 Mbps.- Ensure LB connection tracking and timeouts scale: choose an LB supporting >10k concurrent connections (common).

Network and other considerations- Use 1 Gbps NICs on instances; for high scale use 10 Gbps if aggregating many services.- Monitoring: track CPU per conn, socket usage, retransmits; autoscale by connections or CPU.- TLS termination cost: if LB terminates TLS, move CPU cost to LB; if servers terminate, add CPU budget.- Recommended deployment: 4 app servers (min), 2 LBs (active-active), 1 Gbps network; scale to 5 servers for operational headroom.

Unlock Full Question Bank

Get access to hundreds of Load Balancing and Horizontal Scaling interview questions and detailed answers.

Join thousands of developers preparing for their dream job.