Scalability & Capacity Planning Questions

Analyzing how a system's performance changes as load grows and planning the resources to keep it healthy. Covers horizontal vs vertical scaling, throughput vs latency under load, headroom and saturation, load modeling, and forecasting capacity for expected traffic. Includes identifying the scaling bottleneck that will bind first as demand increases.

EasyTechnical

95 practiced

You are designing instrumentation for a RESTful product search API. Enumerate the essential metrics, logs, and traces you would collect to analyze latency, errors, and throughput. For each metric specify collection frequency, cardinality concerns, suggested dashboards or panels, and example alert thresholds (e.g., p99 > 1s). Indicate which metrics should be aggregated and which should be tagged at high cardinality.

Sample Answer

Situation: Designing instrumentation for a RESTful product search API — goal: detect/triage latency, errors, throughput, and root causes quickly.

Essential metrics (with collection frequency, cardinality guidance, dashboards, alert thresholds):1. Request rate (RPS)- Frequency: 10s- Cardinality: aggregate by service/region (low), tag by endpoint (medium)- Dashboard: time-series of overall RPS + per-endpoint panels- Alert: sustained drop >40% for 5m (possible outage)- Aggregate: yes (rollup per service)

2. Latency histograms (p50/p90/p99) — total request time and component spans (search, db, cache)- Frequency: 10s (histogram buckets)- Cardinality: aggregate overall and per-endpoint; component spans tagged at medium cardinality; avoid high-cardinality user IDs- Dashboard: latency percentiles per endpoint and per region; waterfall breakdown- Alert: p99 > 1s or p90 > 300ms for 5m- Aggregate: store histograms; expose percentile rollups

3. Error rate (%) and counts (4xx/5xx breakdown)- Frequency: 10s- Cardinality: by status code, endpoint, client-app (medium)- Dashboard: error % trend, top endpoints by errors- Alert: 5xx rate > 1% for 5m OR absolute 5xx > 50/min- Aggregate: counts aggregated; tags preserved for drilldown

4. Timeouts & Retries- Frequency: 10s- Cardinality: by downstream (DB/search engine), endpoint (medium)- Dashboard: ratio of timeouts to requests; retry storms- Alert: timeouts increase >200% baseline for 5m

5. Backend dependency latencies & error rates (DB, search index, cache)- Frequency: 10s- Cardinality: dependency name (low), host only if needed (medium)- Dashboard: stacked latency per dependency; correlation with request latency- Alert: any dependency p95 > threshold (e.g., DB p95 > 200ms)

6. Cache hit ratio (e.g., product suggestions)- Frequency: 30s- Cardinality: by cache key pattern or endpoint (medium)- Dashboard: cache hit vs miss and impact on latency- Alert: hit ratio drop >15% sustained 10m

7. Payload size / response bytes- Frequency: 1m- Cardinality: endpoint (low)- Dashboard: avg payload size; detects regression in response bloat- Alert: avg response size > X KB (depends on SLA)

8. Throttling / Rate-limit events- Frequency: 10s- Cardinality: by client-app (high-cardinality tag), but aggregate totals; keep client ID as high-cardinality tag only in logs/traces- Dashboard: top clients tripping throttles- Alert: sudden spike in rate-limited requests > baseline

Logs (structured JSON)- What: request id, timestamp, HTTP method/path, status, latency_ms, user_agent, client_id (or hashed), error stack, upstream dependency traces- Sampling: log all errors (5xx and unexpected 4xx), sample successful requests (1-2%) to control volume- Cardinality: avoid logging raw user IDs or high-cardinality values; hash or bucket when needed- Dashboard: Log-based histograms, top error messages, frequent stack traces- Retention: full error logs 30-90 days; sampled success logs 7-14 days

Traces (distributed tracing)- What: trace per request with spans: ingress, auth, search query, DB, cache, serialization, external calls- Sampling: adaptive tracing — capture 100% of errors, 1-5% of successes, and all traces above latency thresholds (e.g., p99)- Cardinality: include request_id, endpoint, client_id (hashed), region; avoid high-cardinality free-text tags- Dashboard: trace waterfall, flame graphs, slow trace explorer, latency-correlated traces- Alerting integration: link alerts to recent slow/error traces for fast RCA

Tagging and cardinality rules- High-cardinality tags (avoid in metrics): user_id, full session_id, raw query text — keep these in logs/traces only (hashed if needed)- Medium-cardinality tags (allow on some metrics): endpoint/path (use normalized paths, e.g., /search?q -> /search), region, client_app, status_code- Low-cardinality (always allowed): service name, environment, deployment version

Suggested dashboards/panels- Overview: RPS, errors%, p50/p90/p99 latency, success vs error counts- Endpoint heatmap: top endpoints by latency and errors- Dependency panel: per-dependency latency & errors- Capacity panel: CPU, memory, threads, connection pool usage correlated with latency- User-impact panel: SLO compliance (error budget burn, latency SLOs p95/p99), recent alerts

Example SLOs & alert thresholds (adjust to SLA)- Latency SLO: p95 < 200ms, p99 < 1s — Alert when p99 > 1s for 5m- Availability SLO: error rate < 0.5% — Alert when 5xx rate > 1% for 5m- Throughput anomaly: sudden drop >40% for 5m or spike >200% for 2m

Why these choices- Histograms and percentiles capture tail latency which impacts user experience- Aggregated metrics keep storage and query costs manageable; traces/logs provide high-cardinality detail for RCA- Sampling controls volume while preserving signal for errors and outliers

Edge considerations- Normalize endpoints to reduce cardinality (path templates)- Hash or bucket identifiers if you must expose per-user signals in metrics- Use adaptive tracing to surface slow/error requests without tracing everything

This set balances actionable observability with storage and cardinality limits, supporting SLA monitoring and rapid root-cause investigation.

EasyBehavioral

83 practiced

You can only fix one of three issues right now: a bug impacting 5% of revenue, a performance bug that increases p99 latency, or a developer-flakiness slowing deployments. Explain how you would prioritize the fixes and justify your decision to engineering, product, and sales stakeholders. Include metrics and risk considerations in your reasoning.

Sample Answer

Situation: As a Solutions Architect at my last company, we had three concurrent problems: a bug causing revenue loss for ~5% of customers, a performance regression raising p99 latency beyond our 500ms SLO, and intermittent developer-flakiness that delayed deployments by ~30% of release attempts.

Task: I had to pick one fix immediately and justify the choice to engineering, product, and sales.

Action:- I quantified impact: revenue loss = 5% ARR (~$X/month), p99 latency increase affected 40% of user sessions and violated a 500ms SLO (incidents up 3x), deployment flakiness increased lead time by 20% and caused two client demos to be postponed last quarter.- I evaluated risk and customer-experience exposure: revenue loss is direct and measurable; p99 latency harms many users and risks SLAs; flakiness hurts velocity and future deliveries.- I prioritized fixing the p99 performance bug first. Reasoning: - Broad customer impact: latency affects 40% of sessions vs revenue bug affecting a smaller subset. - SLA and churn risk: persistent high p99 increases support tickets and churn probability across many customers; we estimated a 2–4% churn risk if SLA breaches continued. - Mitigation capability: we can apply short-term revenue mitigations (workarounds, targeted refunds) for the revenue bug, and schedule a focused reliability sprint to address developer-flakiness after latency is stabilized.- Communication tailored per stakeholder: - Engineering: technical plan (profiling, hot-path optimization, telemetry, canary rollback), expected work: 2 engineers for 3 days; rollback plan and test coverage to prevent regressions. - Product: customer impact metrics, how fixing latency will reduce support tickets (~30% expected) and preserve SLA NPS; temporary workaround for revenue bug announced to affected customers. - Sales: sellable outcome—stabilized SLO reduces churn risk and secures upcoming renewals; committed remediation timeline for revenue-impacting customers to prevent lost deals.

Result: Stabilizing p99 brought latency back under SLO in our pilot within 48 hours, reducing incident volume by ~60% and preventing projected churn. This bought time to schedule the revenue bug fix in the same sprint and a longer initiative to eliminate deployment flakiness.

This approach balances immediate customer experience, revenue protection via mitigations, and engineering capacity—using data-driven risk and SLO reasoning to justify priorities to all stakeholders.

EasyTechnical

82 practiced

Define backpressure in distributed systems and describe two implementation patterns for HTTP-based services and two patterns for message or streaming systems. For each pattern list advantages, limitations, and scenarios where you would prefer it (for example, HTTP 429, gRPC flow control, token-bucket admission, or queue-depth backpressure).

Sample Answer

Backpressure: a system-level mechanism that prevents producers from overwhelming consumers by signalling or enforcing reduced input rate or dropped work; it preserves stability, predictable latency, and resource bounds in distributed systems.

HTTP-based patterns1) HTTP 429 (client-side throttling)- What: Server returns 429 Too Many Requests (optionally Retry-After).- Advantages: Simple, interoperable, uses standard semantics; easy to implement at proxies or app servers.- Limitations: Reactive (work already rejected), requires clients to implement retry/backoff, no guaranteed fairness.- When to prefer: Public REST APIs, rate-limited endpoints, multi-tenant services where policy enforcement is needed.

2) gRPC flow control (HTTP/2 window-based)- What: Per-stream and connection-level flow-control windows signal available receive buffer; sender pauses when window exhausted.- Advantages: Fine-grained, low-latency, built into protocol; efficient for streaming and back-and-forth RPCs.- Limitations: Requires client/server support (gRPC/H2); complex debugging; doesn't enforce global policies.- When to prefer: High-throughput streaming RPCs, low-latency inter-service communication inside trusted environments.

Message/streaming patterns1) Token-bucket admission (producer-side throttling)- What: Consume tokens to publish/ingest; token refill controls sustained and burst rates.- Advantages: Smooths bursts, predictable throughput control, implementable at gateways or brokers.- Limitations: Requires token coordination if distributed; may delay or drop messages if tokens exhausted.- When to prefer: Ingress gateways, API gateways, services needing burst shaping.

2) Queue-depth backpressure / consumer-driven pull- What: System exposes queue depth or allows consumers to pull N messages; producers/backends slow when queue exceeds thresholds.- Advantages: Simple to reason about, protects consumers and downstream; enables priority or load-shedding policies.- Limitations: Increased latency if pulling; requires observability for thresholds; producing services must react.- When to prefer: Message brokers, batch processing pipelines, systems where durable buffer and consumer control are required.

Trade-offs summary: choose protocol-native mechanisms (gRPC flow control) for low-latency streaming; use HTTP 429 or token-bucket for policy-driven public surfaces; prefer queue-depth/pull where durability and consumer pacing matter.

HardTechnical

137 practiced

A product requires SLOs of p99 latency < 200ms and 99.99% availability. Describe an SLO-driven capacity planning approach: define load profiles, determine headroom and safety margins, propose autoscaling policies and cooldowns, set error budget burn-rate alerts, and design load tests (including edge cases) to validate compliance. Explain how you would present capacity recommendations to stakeholders.

Sample Answer

Requirements & goals:- p99 latency < 200ms, availability ≥ 99.99% (≤52 min/year downtime), define these as SLOs; derive error budget = 0.01% availability loss.

1) Define load profiles- Identify traffic patterns: steady baseline (qps), diurnal peaks, weekly/seasonal, bursty/flash events driven by marketing or client onboarding.- Characterize request mix by endpoint, payload size, CPU/IO cost, and p99 contribution. Produce percentile heatmaps and load-shape templates (baseline, peak, spike).

2) Headroom & safety margins- Start from measured capacity at target SLO (p99 under 200ms). Define N_min = machines needed for baseline with 30–50% headroom to absorb variance and underlying system degradation.- Add redundancy: N_replica >= 2 zones/regions; reserve extra capacity for rolling upgrades and failover (e.g., +1 instance per zone).- Safety margin = max(instantaneous variance, deployment risk) → typically 20–40% depending on SLA.

3) Autoscaling policies & cooldowns- Use multi-dimensional autoscaling: primary on CPU/RPS per instance, secondary on p99 latency and queue length.- Policy: scale-out when sustained p99 > 150ms or CPU>70% or RPS per instance > threshold for 30s; scale-in when metrics under thresholds for 10m.- Cooldowns: short scale-out cooldown (30–60s) for rapid response; longer scale-in cooldown (10–15m) to avoid flapping and allow error-budget recovery.- Warm pools and pre-provisioning for known traffic events.

4) Error budget burn-rate alerts- Track rolling 30d and 7d burn rates. Alert tiers: - NOTICE: burn-rate > 1.0 for 1h → warn engineering. - ACTION: burn-rate > 4.0 sustained 1h → pause risky releases, increase capacity. - CRITICAL: burn-rate > 10.0 → trigger incident playbook.- Tie automated mitigations: scale to safe capacity and rollback releases if correlated to deploys.

5) Load testing to validate- Test types: steady-state at 1.2x baseline, peak at 1.5–2x, spike tests (10x short burst), chaos (instance/network failures), and long-duration soak (48–72h).- Measure p99, error rates, queue growth, GC/latency tail, and failover time. Include region failover and degraded dependency simulation.- Edge cases: large payloads, authentication storms, DB contention, slow downstream caches, cold-starts, and mixed traffic shapes.

6) Presenting recommendations to stakeholders- Deliver a concise deck: business impact (SLOs → user pain & revenue risk), capacity plan (baseline N, headroom %, redundancy), cost vs risk trade-off, autoscaling policy summary, error-budget governance, and testing results with pass/fail.- Provide three options: conservative (low cost, higher risk), recommended (balanced), aggressive (high cost, minimal risk) with expected availability, cost delta, and operational implications.- Attach runbook excerpts (alerts, rollback triggers), key dashboard links, and a phased rollout plan with guardrails and success metrics.

MediumTechnical

67 practiced

Explain how rate limiting and backpressure can be combined in a distributed event ingestion pipeline to prevent downstream overload while preserving acceptable client experience. Include token-bucket parameters, retry policies, upstream pushback mechanisms, and how to treat bursty traffic and priority tenants.

Sample Answer

Start with clear goals: protect downstream capacity, keep high-priority tenants responsive, and give clients predictable behavior. Combine token-bucket rate limiting at ingress with backpressure signals upstream and client-friendly retry rules.

Ingress token-bucket: per-tenant buckets with refill rate R (tokens/sec) = steady allowed rate, and bucket size B (tokens) = allowable burst. Choose B ≈ R * burst_window (e.g., 10s) to absorb short spikes. Use weighted fair queues or weighted-token-buckets so priority tenants get larger R or weight W. Also enforce a global pool to protect shared resources.

When downstream queues exceed thresholds (e.g., queue_length > Q_high or processing_latency > L_high), apply backpressure:- Immediate soft-throttle: reduce per-tenant refill rate R dynamically (controller lowers R or injects tokens slower).- Hard pushback: return 429/RESOURCE_EXHAUSTED with Retry-After and a machine-readable reason; for gRPC use flow-control WINDOW updates or GOAWAY-like signals.

Retry policy for clients:- Encourage exponential backoff with jitter (base 100–500ms, cap 5–60s), respect Retry-After header. Limit total retries and use idempotency keys to avoid duplicate processing.

Handling bursty traffic:- Allow bursts via B but bound with admission control: if bucket depleted and downstream under stress, place requests into prioritized waiting queues with timeout; low-priority requests get dropped or 429 sooner.- Use short-term spillover storage (durable queue) for high-priority events when temporary downstream slowdown occurs.

Fairness & priority:- Implement hierarchical buckets: tenant -> tenant-class -> global. Apply minimum guaranteed R for SLA tiers and maximum caps. Use deficit-round-robin or weighted fair queuing to allocate capacity under contention.

Operational considerations:- Fast controller for dynamic throttling, conservative smoothing (moving averages over 10–30s) to avoid oscillation.- Telemetry: per-tenant token consumption, 429 rate, queue latency, retry counts.- Test with chaos/traffic replay to tune R, B, thresholds.

This combination preserves client experience by allowing short bursts, giving clear retry guidance, prioritizing critical tenants, and avoiding cascading failures through explicit upstream pushback.

Unlock Full Question Bank

Get access to hundreds of Scalability & Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.