Load Balancing and Horizontal Scaling Questions

Covers principles and mechanisms for distributing traffic and scaling services horizontally. Includes load balancing algorithms such as round robin, least connections, and consistent hashing; health checks, connection draining, and sticky sessions; and session management strategies for stateless and stateful services. Explains when to scale horizontally versus vertically, capacity planning, and trade offs of each approach. Also includes infrastructure level autoscaling concepts such as auto scaling groups, launch templates, target tracking and step scaling policies, and how load balancers and autoscaling interact to absorb traffic spikes. Reviews different load balancer types and selection criteria, integration with service discovery, and operational concerns for maintaining availability and performance at scale.

MediumTechnical

69 practiced

Describe, step-by-step, how a typical cloud load balancer and autoscaling group interact during a sudden traffic spike that increases requests/sec by 5x in one minute. Explain registration/deregistration, health checks, scale-out latency, connection queueing, and short-term mitigation options to avoid degraded user experience.

Sample Answer

Situation: Traffic suddenly spikes 5x in one minute to a service behind a cloud load balancer (LB) and an autoscaling group (ASG).

Step‑by‑step interaction (what happens and why):1. LB receives 5x requests/sec and begins distributing new connections to existing healthy targets. If per‑instance capacity is exceeded, request queueing or TCP backlog grows at the LB or nodes.2. LB health checks continue — healthy targets remain in rotation; overloaded instances may start failing checks (latency/timeouts) and be marked unhealthy and deregistered, reducing pool size and worsening load.3. ASG’s scaling policies observe metrics (CPU, RPS, custom latency). A target‑tracking or step policy triggers scale‑out: ASG requests new instances/containers from the cloud provider.4. Registration: once instances pass initial health (agent ready, app listening), they register with the LB (or LB autodiscovers). Only after passing LB health checks do they receive traffic.5. Scale‑out latency: includes instance launch time (VM/container startup), app initialization, JIT/connection pools, and LB health‑check warm‑up. This can be tens of seconds to minutes—creating a shortfall period.6. Deregistration/connection draining: when instances are removed (unhealthy or scaled in), LB drains existing connections per configured timeout so in‑flight requests finish.

Short‑term mitigation options to avoid degraded UX:- Increase capacity at LB layer: raise max concurrent connections or add LB nodes if configurable.- Reduce per‑instance startup and warm‑up: use pre‑baked images, lightweight containers, or warm pools to cut scale latency.- Use proactive (predictive) scaling or step scaling: trigger earlier scale‑outs using RPS or queue length metrics instead of CPU.- Enable connection queueing and circuit breakers: queue requests at edge and return quick 429s for overload to signal clients to back off.- Graceful degradation & caching: serve stale cache, reduce feature load, disable nonessential work.- Tune health checks and deregistration: lengthen health‑check thresholds slightly to avoid premature removal; enable slow‑start/warm‑up on LB so new instances ramp traffic gradually.- Short bursts: serve requests from CDN, edge cache, or API gateway throttling to absorb spike.- Emergency rules: temporarily increase ASG max size, enable surge capacity or spot/fleet fallback.

Key tradeoffs and recommendations:- Aggressive health checks remove sick nodes fast but can intensify capacity loss—balance probe sensitivity.- Warm pools and predictive scaling reduce user impact but cost more.- Measure scale‑out latency and set autoscale trigger thresholds earlier (e.g., at 60–70% of instance RPS) and use request‑rate metrics for quicker detection.

Result: Combining faster instance start (warm pools), smarter metrics for earlier scaling, LB slow‑start, edge caching/throttling and temporary capacity increases minimizes user-visible errors during sudden spikes.

HardTechnical

63 practiced

Your organization wants to reduce cost of load balancing and autoscaling while keeping SLOs. Propose three concrete strategies (e.g., using spot instances, right-sizing instances, using fewer larger instances, request batching, API gateway caching). For each strategy, list risks and mitigations.

Sample Answer

Strategy 1 — Use spot/preemptible instances for non-critical and batch workloads- What: Move stateless workers, background jobs, and scale-to-zero pools to spot VMs (or mixed node pools in k8s) while keeping a small on-demand baseline for steady-state.- Benefits: Large cost reduction (30–70%) on compute used during autoscaling spikes.- Risks & mitigations: - Risk: Preemption causes job failures or capacity gaps → Mitigate: checkpointing/retry logic, idempotent tasks, short-lived task queues (e.g., durable queue like SQS), and diversify across zones/providers. - Risk: Thundering rebuilds on mass eviction → Mitigate: gradual scale-up policies, PodDisruptionBudget/priorityClass in k8s, and maintain a warm on-demand buffer sized by SLO risk budget.

Strategy 2 — Right-size instances and use fewer larger instances for steady base capacity- What: Analyze telemetry (CPU, memory, network, I/O) to downsize oversized instances and consider consolidating into fewer larger instances where appropriate.- Benefits: Reduce per-instance load-balancer and autoscaling churn, lower instance-hours and LB rules cost.- Risks & mitigations: - Risk: Over-consolidation increases blast radius and impacts availability → Mitigate: use fault domains (AZs), autoscaling groups across AZs, and reserve capacity with redundancy (n+1). - Risk: Hotspots or noisy neighbors on larger instances → Mitigate: resource limits/requests, cgroups/containers, and observability alerts for elevated tail latency.

Strategy 3 — Request batching and API gateway caching for high QPS endpoints- What: Implement batching on backend services and enable caching (edge + API gateway + in-app TTL) for idempotent reads; add a cache invalidation strategy.- Benefits: Lowers backend compute and load-balancer throughput, reducing scale-up events and cost.- Risks & mitigations: - Risk: Increased tail latency for batched requests or stale cache responses → Mitigate: set SLO-aligned batch timeouts and size limits; for caching, set conservative TTLs, use cache-control headers and cache-busting on writes. - Risk: Complexity in client change and correctness → Mitigate: provide client libraries, gradual rollout, feature flags, and end-to-end tests to validate semantics.

Operational controls across all strategies:- Use SLO-driven error budgets to decide how aggressive cost cuts can be.- Implement dashboards and burn-rate alerts for latency/errors during rollout.- Automate rollbacks and canary deployments to limit user impact.

MediumTechnical

64 practiced

Design an ingress/load balancing architecture for a Kubernetes cluster that must support 100k RPS and long-lived WebSocket connections. Include components (external LB type, ingress controller, service type, node sizing), connection draining strategy for rolling nodes, and how you would test at scale.

Sample Answer

Situation: You need a Kubernetes ingress/load-balancing architecture that can handle 100k RPS plus many long-lived WebSocket connections. Below is a practical, SRE-focused design with components, node sizing, draining strategy, and scale-testing plan.

Design summary- External LB: Use a Layer‑4 Network Load Balancer (NLB) or equivalent (e.g., AWS NLB, GCP Network LB). Reason: preserves TCP connections, very high connection throughput, low latency and native support for connection stickiness/timeouts required by long-lived WebSockets.- Ingress controller: Deploy Envoy (as ingress gateway) or NGINX/Envoy-based Ingress with TCP passthrough support. Envoy is preferred for performance, connection management, observability and seamless WebSocket handling.- Service type: Use NodePort/DaemonSet fronting Envoy (or hostPort) with externalTrafficPolicy: Local to preserve client source IPs and avoid hairpin. Alternatively use a Service of type LoadBalancer per cluster in environments that support NLBs directly.- Topology: NLB → Envoy (DaemonSet on every node or dedicated gateway nodes) → Kubernetes ClusterIPs → backend Pods (Deployment).- Sticky/affinity: Prefer sticky by connection (TCP) at NLB layer for session affinity; application-level reconnection logic handles failover.

Node sizing & capacity planning- Separate gateway nodes (2–3 AZs) and worker nodes. Gateway nodes sized for connection fanout and TLS termination: e.g., m5.4xlarge or c6i.4xlarge (16 vCPU, 64GB) depending on TLS CPU. Each gateway node can sustain tens of thousands of concurrent connections—benchmark first.- Backend nodes sized by RPS and CPU per request (use profiling). For 100k RPS, estimate per-request CPU and memory, then compute pods = (100k * p95 CPU per request) / CPU per pod. Start with autoscaling (HPA/Cluster Autoscaler) and capacity buffer (25–50%).- Use vertical pod autoscaler for Envoy resource tuning; reserve ephemeral ports and ulimits for high connection counts.

Connection draining / rolling nodes- At node termination: - Set NLB deregistration delay to a generous value (e.g., 300s) matching expected drain time. - On pod SIGTERM: - Disable readiness (make pod not ready immediately) so Service stops sending new connections. - Implement preStop hook that: - Stops accepting new WebSocket upgrades - Waits for active connections to finish or for graceful timeout (e.g., 5–10m) - If using Envoy, use admin endpoint to drain listeners (POST /drain_listeners) and set connection_close_timeout. - Only after drain complete, exit process. - Use node’s kubelet –eviction and cordon prior to termination; coordinate with cluster autoscaler. - Force kill after a conservative hard timeout.

Observability & SLOs- Collect metrics: connection counts, connection lifetime, RPS, p99 latency, 5xx rates per layer (NLB, Envoy, app).- Traces for upgrade path and reconnection behavior; alert on connection saturation, high reconnect rates.- SLOs: availability & latency targets for both short requests and WebSocket functionality.

Testing at scale- Functional & load testing plan: - Stage: run full-scale tests in a staging cluster that mirrors production AZs and node types. - Tools: - For HTTP RPS: k6, vegeta, wrk2 to generate 100k RPS distributed across many generators. - For WebSocket: custom load generators (Gatling/TSUNG or k6 websocket mode, or a scaled Go program using goroutines) to create thousands–100ks of long-lived connections and perform occasional messages. - Use containerized generators spread across separate worker VMs to avoid client bottlenecks. - Tests to run: - Ramp tests: ramp RPS and concurrent connections to target while monitoring tail latency and failures. - Soak tests: maintain target load for hours to find leaks and resource exhaustion. - Failure injection: kill gateway pods/nodes during load, simulate NLB failover, test reconnection behavior. - TLS offload vs pass-through comparison if applicable. - Validate metrics: error budgets, connection acceptance rate, reconnect frequency, CPU/NET saturation.

Alternatives & trade-offs- Using L7 ALB/IngressController (e.g., AWS ALB) simplifies routing and TLS but ALBs can struggle with extreme connection counts and long-lived sockets—only choose if ALB supports websockets at required scale and connection limits are acceptable.- HostPort/DaemonSet reduces a hop but increases node management complexity.

Key operational tips- Pre-warm NLB targets and TCP ephemeral port tuning on nodes (sysctl fs.file-max, net.core.somaxconn).- Tune Envoy: listener buffer limits, connection limit per worker, worker threads equal to vCPUs.- Use auto-scaling for both gateway and backend tiers and set conservative pod disruption budgets.

This design focuses on L4 fronting (NLB) + high-performance Envoy ingress, explicit draining, capacity planning, and realistic large-scale testing to reach 100k RPS with stable long-lived WebSocket support.

HardTechnical

64 practiced

You led an incident where autoscaling didn’t trigger and the load balancer overloaded healthy backends, causing a multi-hour outage. You are asked to prepare and lead the post-incident review and follow-up. Describe the structure of your postmortem, stakeholders to involve, actionable remediation items (with owners and deadlines), and how to prevent recurrence while communicating with customers.

Sample Answer

Situation: A production outage lasted multiple hours because autoscaling failed to trigger and the load balancer overloaded healthy backends. I led the post-incident review to identify root causes, assign remediation, and restore customer trust.

Postmortem structure:- Executive summary (impact, timeline, customer-facing statement)- Timeline of events (detection → mitigation → resolution) with timestamps- Root cause analysis (technical root cause + contributing factors)- Detection & response analysis (alerts, runbook use, communication gaps)- Blast radius & impact metrics (users affected, error rates, revenue/SLAs)- Action items (owner, priority, deadline, verification criteria)- Follow-up and lessons learned- Appendices (logs, graphs, runbook excerpts)

Stakeholders to involve:- SRE leads (incident commander, tooling owner)- Backend and platform engineers (autoscaler, LB)- Product owner / PM- Customer support & communications- Security & compliance (if relevant)- Engineering manager(s) for resourcing and deadlines

Actionable remediation items (examples with owners & deadlines):1) Fix autoscaler bug and add unit + integration tests — Owner: Platform Eng, Due: 2 weeks. Verification: CI test coverage + staged canary.2) Add LB backpressure and circuit-breaker to avoid overloading healthy hosts — Owner: Backend Eng, Due: 4 weeks. Verification: load test showing degraded instances are removed.3) Create/repair alerting for scaling failures (rate of scaling ops, pending scale requests) — Owner: Observability SRE, Due: 1 week. Verification: Test alert in staging + runbook entry.4) Update runbooks with escalation path and auto-remediation playbooks — Owner: Incident Commander / SRE Docs, Due: 1 week. Verification: Runbook walkthrough exercise.5) Implement automated chaos/scale tests in CI to catch regressions — Owner: Reliability Eng, Due: 6 weeks. Verification: CI job passes and simulated failure results logged.6) Postmortem review session with execs and affected teams; publish customer-facing FAQ — Owner: Incident Lead, Due: 3 business days.

Prevent recurrence (process & technical):- Define SLOs and error budget triggers tied to autoscaling behavior.- Add synthetic load tests and scheduled chaos experiments targeting scaling and LB behavior.- Improve observability: metrics for queued scale requests, LB per-backend queue depth, tail latency.- Enforce code review checklist for platform changes that affect scaling or LB logic.- Quarterly incident response drills and runbook tabletop exercises.

Customer communication plan:- Within 3 hours: initial acknowledgement with estimated investigation timeline from Support/Comms.- Within 24 hours: incident summary and mitigation steps taken.- Within 72 hours: detailed postmortem link + FAQ and any compensation/SLR steps if applicable.- Offer direct outreach to top affected customers and a webinar/Q&A if impact large.

This approach couples technical fixes with people/process changes, measurable verification, and transparent customer communication to rebuild reliability and trust.

MediumTechnical

70 practiced

Explain the benefits and drawbacks of cross-zone (or cross-availability-zone) load balancing. How does it affect latency, cost, and capacity utilization? When might you disable cross-zone balancing and what alternative would you implement to preserve availability?

Unlock Full Question Bank

Get access to hundreds of Load Balancing and Horizontal Scaling interview questions and detailed answers.

Join thousands of developers preparing for their dream job.