Scalability Analysis and Bottleneck Identification Questions

Techniques for analyzing existing systems to find and prioritize bottlenecks and to validate scaling hypotheses. Topics include profiling and benchmarking strategies instrumentation and monitoring of latency throughput error rates and resource utilization; identification of common bottlenecks such as database write throughput central processing unit saturation memory pressure disk input output limits and network bandwidth constraints; designing experiments and load tests to reproduce issues and validate mitigations; proposing incremental fixes such as caching partitioning asynchronous processing or connection pooling; and measuring impact with clear metrics and iteration. Interviewers will probe the candidate on moving from observations to root cause and on designing low risk experiments to validate improvements.

EasyBehavioral

62 practiced

You can only fix one of three issues right now: a bug impacting 5% of revenue, a performance bug that increases p99 latency, or a developer-flakiness slowing deployments. Explain how you would prioritize the fixes and justify your decision to engineering, product, and sales stakeholders. Include metrics and risk considerations in your reasoning.

Sample Answer

Situation: As a Solutions Architect at my last company, we had three concurrent problems: a bug causing revenue loss for ~5% of customers, a performance regression raising p99 latency beyond our 500ms SLO, and intermittent developer-flakiness that delayed deployments by ~30% of release attempts.

Task: I had to pick one fix immediately and justify the choice to engineering, product, and sales.

Action:- I quantified impact: revenue loss = 5% ARR (~$X/month), p99 latency increase affected 40% of user sessions and violated a 500ms SLO (incidents up 3x), deployment flakiness increased lead time by 20% and caused two client demos to be postponed last quarter.- I evaluated risk and customer-experience exposure: revenue loss is direct and measurable; p99 latency harms many users and risks SLAs; flakiness hurts velocity and future deliveries.- I prioritized fixing the p99 performance bug first. Reasoning: - Broad customer impact: latency affects 40% of sessions vs revenue bug affecting a smaller subset. - SLA and churn risk: persistent high p99 increases support tickets and churn probability across many customers; we estimated a 2–4% churn risk if SLA breaches continued. - Mitigation capability: we can apply short-term revenue mitigations (workarounds, targeted refunds) for the revenue bug, and schedule a focused reliability sprint to address developer-flakiness after latency is stabilized.- Communication tailored per stakeholder: - Engineering: technical plan (profiling, hot-path optimization, telemetry, canary rollback), expected work: 2 engineers for 3 days; rollback plan and test coverage to prevent regressions. - Product: customer impact metrics, how fixing latency will reduce support tickets (~30% expected) and preserve SLA NPS; temporary workaround for revenue bug announced to affected customers. - Sales: sellable outcome—stabilized SLO reduces churn risk and secures upcoming renewals; committed remediation timeline for revenue-impacting customers to prevent lost deals.

Result: Stabilizing p99 brought latency back under SLO in our pilot within 48 hours, reducing incident volume by ~60% and preventing projected churn. This bought time to schedule the revenue bug fix in the same sprint and a longer initiative to eliminate deployment flakiness.

This approach balances immediate customer experience, revenue protection via mitigations, and engineering capacity—using data-driven risk and SLO reasoning to justify priorities to all stakeholders.

MediumSystem Design

56 practiced

Given a MySQL table 'orders(order_id PK, user_id, amount, created_at)' receiving 50k writes/sec and exhibiting hot partitions by user_id, propose partitioning and sharding options such as range partition by date, hash partition by order_id, and user-based sharding. For each option explain the effect on write throughput, common query patterns, migration strategy with minimal downtime, and implications for cross-shard transactions.

Sample Answer

Requirements recap: 50k writes/sec, hot partitions by user_id, queries include recent-orders by date, user order history, and order lookup by order_id. Need minimal downtime migration and reasonable cross-shard transaction behavior.

1) Range partition by date (MySQL native RANGE on created_at)- Write throughput: improves IO locality for recent data (writes concentrated to newest partition) but doesn't solve user hot-spot if many writes are from same user — newer partitions still hot but smaller index/IO.- Common queries: fast range scans for recent orders, efficient TTL/archival (drop old partitions).- Migration: create new partitioned table and use online table copy (pt-online-schema-change or MySQL ALTER TABLE ... ALGORITHM=INPLACE if supported). Use dual-writes to new table and backfill in background; cutover when caught up.- Cross-shard/transaction: single-table partitioning stays in same instance so transactions unaffected.

2) Hash partition by order_id (MySQL PARTITION BY HASH(order_id))- Write throughput: spreads writes evenly across partitions on same server, reducing per-partition lock contention but still single MySQL instance limits (no cross-host scale).- Common queries: efficient single-order lookup; range/date queries still require scanning partitions unless secondary indexes or partition pruning by date expression (limited).- Migration: create new partitioned table and do online copy/backfill; or use CREATE TABLE ... PARTITION and swap with RENAME TABLE using online tool to avoid downtime.- Cross-shard/transaction: still single instance, so transactions remain simple.

3) User-based sharding (application-level shard by user_id -> multiple MySQL instances)- Write throughput: horizontally scales writes linearly by adding shards; resolves user hot-spot by spreading noisy users across shards (need hash/mod strategy or consistent-hash with shard mapping).- Common queries: user-history and single-order lookups are local to a shard (fast). Global queries (recent orders across users, analytics) require fan-out / aggregation or a separate OLAP pipeline.- Migration: map ranges/hashed user_id to new shard; use dual-write or write-forwarder at app/API gateway to write to both old and new during cutover; run background backfill of historical rows (per-user) with idempotent upserts; Reroute once caught up. Use consistent-hash + routing table to minimize remapping.- Cross-shard transactions: expensive — need distributed transactions or two-phase commit; better to design for eventual consistency, use compensating actions, or colocate related users/orders. Avoid cross-shard transactions where possible.

Recommendation: Start with user-based sharding for write scale and hotspot resolution. Complement with date-range partitioning within each shard for fast retention/maintenance. Use a migration plan: add new shards, dual-write + backfill per-user, update routing, then retire old shard. Maintain global secondary index/stream (Kafka) for cross-shard queries and analytics to avoid costly distributed transactions.

HardTechnical

107 practiced

A product requires SLOs of p99 latency < 200ms and 99.99% availability. Describe an SLO-driven capacity planning approach: define load profiles, determine headroom and safety margins, propose autoscaling policies and cooldowns, set error budget burn-rate alerts, and design load tests (including edge cases) to validate compliance. Explain how you would present capacity recommendations to stakeholders.

Sample Answer

Requirements & goals:- p99 latency < 200ms, availability ≥ 99.99% (≤52 min/year downtime), define these as SLOs; derive error budget = 0.01% availability loss.

1) Define load profiles- Identify traffic patterns: steady baseline (qps), diurnal peaks, weekly/seasonal, bursty/flash events driven by marketing or client onboarding.- Characterize request mix by endpoint, payload size, CPU/IO cost, and p99 contribution. Produce percentile heatmaps and load-shape templates (baseline, peak, spike).

2) Headroom & safety margins- Start from measured capacity at target SLO (p99 under 200ms). Define N_min = machines needed for baseline with 30–50% headroom to absorb variance and underlying system degradation.- Add redundancy: N_replica >= 2 zones/regions; reserve extra capacity for rolling upgrades and failover (e.g., +1 instance per zone).- Safety margin = max(instantaneous variance, deployment risk) → typically 20–40% depending on SLA.

3) Autoscaling policies & cooldowns- Use multi-dimensional autoscaling: primary on CPU/RPS per instance, secondary on p99 latency and queue length.- Policy: scale-out when sustained p99 > 150ms or CPU>70% or RPS per instance > threshold for 30s; scale-in when metrics under thresholds for 10m.- Cooldowns: short scale-out cooldown (30–60s) for rapid response; longer scale-in cooldown (10–15m) to avoid flapping and allow error-budget recovery.- Warm pools and pre-provisioning for known traffic events.

4) Error budget burn-rate alerts- Track rolling 30d and 7d burn rates. Alert tiers: - NOTICE: burn-rate > 1.0 for 1h → warn engineering. - ACTION: burn-rate > 4.0 sustained 1h → pause risky releases, increase capacity. - CRITICAL: burn-rate > 10.0 → trigger incident playbook.- Tie automated mitigations: scale to safe capacity and rollback releases if correlated to deploys.

5) Load testing to validate- Test types: steady-state at 1.2x baseline, peak at 1.5–2x, spike tests (10x short burst), chaos (instance/network failures), and long-duration soak (48–72h).- Measure p99, error rates, queue growth, GC/latency tail, and failover time. Include region failover and degraded dependency simulation.- Edge cases: large payloads, authentication storms, DB contention, slow downstream caches, cold-starts, and mixed traffic shapes.

6) Presenting recommendations to stakeholders- Deliver a concise deck: business impact (SLOs → user pain & revenue risk), capacity plan (baseline N, headroom %, redundancy), cost vs risk trade-off, autoscaling policy summary, error-budget governance, and testing results with pass/fail.- Provide three options: conservative (low cost, higher risk), recommended (balanced), aggressive (high cost, minimal risk) with expected availability, cost delta, and operational implications.- Attach runbook excerpts (alerts, rollback triggers), key dashboard links, and a phased rollout plan with guardrails and success metrics.

MediumTechnical

75 practiced

Design a low-risk canary experiment to validate that switching to a pooled DB driver reduces latency under production traffic. Specify canary rollout percentage, metrics to compare (latency percentiles, DB connection counts, error rates), monitoring windows, automatic rollback criteria, and how to control for traffic differences between canary and baseline.

Sample Answer

Situation: We need a low-risk canary to validate that replacing the DB driver with a pooled-driver reduces latency under real production load without destabilizing services.

Plan (rollout and duration)- Start very small and increase only after clear signals: - 0.5% for 30 minutes - 2% for 1 hour - 10% for 4 hours - 25% for 24 hours - 50% for 48 hours before full rollout- Use progressive automated promotion that requires all checks to pass at each stage.

Metrics to collect and compare (baseline vs canary, same time window)- Latency percentiles for end-to-end requests: p50, p95, p99, and max.- DB interaction latency: per-query median and p95; connection acquisition/wait time.- DB connection counts: open connections, active vs idle, and connection churn rate.- Error rates: application errors, DB errors/timeouts, transaction aborts.- Throughput (RPS) and success rate.- Resource utilization: app JVM/CPU, DB CPU/memory, connection pool saturation metrics.- Business KPIs: page render time, checkout completion if applicable.

Monitoring windows and statistical checks- Short-term checks at each stage: continuous 5–15 minute aggregation with alerts for abrupt regressions.- Stability windows: require no regression for the full stage duration (listed above) with aggregated hourly stats.- Use rolling windows (5m, 30m, 1h) and compare to simultaneously collected baseline to avoid time-of-day bias.- Run statistical tests on distributions (e.g., Mann–Whitney for latency distributions) and require significance (p < 0.05) before promotion.

Automatic rollback criteria (fail-fast)Trigger immediate rollback if any of:- p95 latency degrades by ≥ 20% and p99 degrades by ≥ 10% compared to baseline with statistical significance.- Error rate increases by absolute ≥ 0.5% or relative ≥ 3x baseline.- Connection pool exhaustion: active connections equals pool size for > 5 minutes or DB reports connection errors/timeouts > threshold.- Any database-level alerts (e.g., rising locked transactions, replication lag) correlated with canary traffic.- Resource spikes (DB CPU > 85% sustained for 5+ minutes) attributable to canary.Rollback implementation: orchestration via feature flag / traffic router that can cut canary to 0% within <1 minute and trigger automated incident runbook.

Controlling for traffic differences- Deterministic user hashing: route identical user segments to baseline and canary across stages to balance user-agent / geography / client versions.- Traffic mirroring for query-level comparison: mirror a sampled subset of baseline traffic to a non-impacting pooled-driver sandbox to compare per-query latencies without affecting production DB.- Match on request types: ensure canary receives same mix of read/write and heavy/light queries (use sampling or weighted routing).- Time-aligned comparisons: compare baseline and canary metrics on the same time slices to remove diurnal effects.- Use query fingerprinting to compare latency per query pattern (so reductions aren’t due to a shift in query mix).

Other controls and best practices- Add extra telemetry: tag traces with driver-version, connection-id, pool-wait times; use distributed tracing to verify tail latency source.- Start with read-only / low-risk endpoints if possible before full-read/write traffic.- Run a chaos-style rollback dry-run to verify fast rollback path.- Communicate plan with DB admins and on-call teams; schedule low-traffic windows for risky stages.

This plan minimizes blast radius, uses statistical comparison and per-query controls to ensure observed latency improvements are due to the pooled driver and not traffic differences, and provides clear automated rollback triggers.

MediumTechnical

52 practiced

Explain how you would collect and use flamegraphs to find CPU hotspots for a JVM service in production with minimal overhead. Describe tools and steps (for example async-profiler -> perf -> flamegraph.pl), sampling strategy, safe ways to capture stacks in production, and how to interpret wide blocks versus deep stacks in a flamegraph to prioritize fixes.

Unlock Full Question Bank

Get access to hundreds of Scalability Analysis and Bottleneck Identification interview questions and detailed answers.

Join thousands of developers preparing for their dream job.