Scalability & Capacity Planning Questions

Analyzing how a system's performance changes as load grows and planning the resources to keep it healthy. Covers horizontal vs vertical scaling, throughput vs latency under load, headroom and saturation, load modeling, and forecasting capacity for expected traffic. Includes identifying the scaling bottleneck that will bind first as demand increases.

EasyTechnical

94 practiced

You observe a sudden increase in error rate on a critical endpoint. List the first 6 steps you would take to triage this production incident to identify whether the cause is code, infra, or external dependency related. Make your answer concrete and ordered by priority.

MediumTechnical

67 practiced

A service shows high disk write latency. Describe an experiment to determine whether the bottleneck is due to slow disks, increased fsync frequency, or database configuration (e.g., WAL settings). Include safe steps for production validation.

Sample Answer

Goal: distinguish whether high write latency comes from (A) slow physical disks, (B) increased fsync frequency from the app/DB, or (C) DB config (WAL/checkpoint settings). Experiment plan with safe production validation:

1) Prepare & safety- Take full backup and ensure recent snapshots. Prefer running experiments on a standby/replica or a copy of the volume. If you must test on primary, schedule a low-traffic window and notify stakeholders.- Put monitoring/alerting in place: iostat/dstat, node_exporter, DB metrics, latency SLOs. Record baseline for 10–30 minutes.

2) Baseline metrics (non-invasive)- Observe OS/disk: iostat -x 1 30 (awaited await, svctm, util%), cat /proc/diskstats.- Observe queue depth and latency: sudo blktrace/blkparse or use perf tools.- DB metrics: WAL write/sync rate, checkpoint activity, checkpoint_duration, wal_fsync, wal_write, synchronous_commit events (Postgres: SELECT * FROM pg_stat_bgwriter; SHOW wal_level; SHOW synchronous_commit;).Interpretation: If device await and avgqu-sz are high -> disk slow/overutilized. If device util low but many fsyncs and DB WAL throughput high -> fsync frequency.

3) Isolate fsync frequency- On a safe replica, enable tracing of fsync calls from DB process: sudo strace -e trace=fsync -p <pid> or use perftrace to count fsyncs/s. Measure fsyncs/sec vs write latency.- Simulate increased fsync: temporarily change app/DB setting on replica to batch more writes (e.g., Postgres: set synchronous_commit = off or modify wal_sync_method) and observe change. Expect: if latency drops dramatically, fsync frequency/config is culprit.Safety: do NOT disable fsync on primary; use a replica or test instance.

4) Disk latency causation test- Run controlled write microbench on same device (on replica or maintenance host): fio --name=latencytest --filename=/mount/targetfile --direct=1 --sync=1 --rw=write --bs=4k --size=1G --iodepth=1- Compare fio measured latencies to observed DB write latencies. If fio shows high latency, disk or host I/O path is root cause (device, controller, virtualization layer, shared noisy neighbor).

5) DB configuration checks- Review DB checkpoint/WAL params (Postgres examples): checkpoint_timeout, checkpoint_completion_target, wal_buffers, wal_writer_delay, wal_sync_method, synchronous_commit, fsync. Temporarily increase checkpoint_completion_target or adjust checkpoint_timeout on replica to reduce checkpoint bursts and watch latency.- If changing DB config reduces latency on replica, config is likely contributor.

6) Correlate & conclude- Correlate time series: fsyncs/sec, WAL bytes/sec, iostat await, queue depth, checkpoint events, tail application latency.- Root cause patterns: - Disk-limited: fio and iostat high await, all write sources slow. - Fsync frequency: many fsyncs/sec, low device util, latency tightly correlated with fsync bursts. - DB config: checkpoint bursts, high wal write spikes, tunable via checkpoint/wal params.

7) Remediation steps (safe)- If disk: increase IOPS/throughput (better disks, RAID tuning, NVMe), move WAL to faster disk, tune queue_depth, or reduce cross-tenant noise.- If fsync frequency: batch writes at app-level, change DB settings on a controlled rollout (replica test -> promote), avoid disabling fsync on primary.- If DB config: tune checkpoint parameters, wal_buffers, and implement gradual change with monitoring.

Key principles: always test on replica or during low traffic, take backups, change one variable at a time, and correlate multiple telemetry sources.

MediumTechnical

98 practiced

Explain how network bandwidth constraints can manifest at different layers (application, OS, infrastructure). Given 1 Gbps NICs and observed packet drops, list diagnostic commands/tools and the order you would run them to isolate whether the problem is on the host, network, or remote dependency.

Sample Answer

Network bandwidth limits and packet loss can appear at multiple layers:

- Application layer: excessive concurrency, small writes, synchronous blocking, or protocol misuse (e.g., chatty RPCs) causes throughput collapse or retransmits even if NIC capacity is fine.- OS/Host layer: kernel networking limits (socket buffers, tx/rx ring overflow), CPU saturation, driver bugs, NIC offload issues, or misconfigured qdiscs/iptables causing drops.- Infrastructure layer: switch/port errors, oversubscription, faulty cables, duplex/MTU mismatch, or upstream congestion shaping.

Ordered diagnostics (quick → deeper), with typical Linux commands and what to look for:

1. Basic reachability & perf- ping -c 10 <remote> (latency, loss)- traceroute/tracepath <remote> (path hops, where loss appears)

2. Measure throughput & retransmits- iperf3 -c <remote> (saturate link; compare host-to-host)- curl/wget or application-level test to reproduce

3. Check TCP stats and retransmits- ss -tpi or netstat -s | grep -i retrans (retransmits, congestion)- sudo cat /proc/net/snmp{,6} for TCP counters

4. Inspect NIC and driver on host- ethtool -S eth0 (driver stats, drops, ring errors)- ethtool eth0 (speed/duplex/auto-neg/TSO/GSO offload)- sudo ethtool -k eth0 (offload settings)

5. Kernel queues and drops- ip -s link show dev eth0 (rx/tx dropped)- tc -s qdisc show dev eth0 (qdisc drops)- watch -n1 "ss -s; ip -s link show eth0"

6. CPU and system limits- top / vmstat / iostat (CPU, steal)- pidstat -d -p <app_pid> (I/O/CPU per process)

7. Capture packets to localize- sudo tcpdump -i eth0 host <remote> and inspect retransmits, dup ACKs- wireshark/tshark if needed; look for retransmit timing and MSS/MTU issues

8. Network path and infrastructure checks (with network team)- show interface counters on switches, check port errors, duplex/CRC- verify ACLs, policing/shaping, and congestion on aggregation links- SPAN capture at switch if host capture inconclusive

9. Remote dependency verification- Run iperf3 from remote->other remote to rule out remote host NIC limits- Check remote host’s ethtool/ss/tc as above

Reasoning: start with cheap tests (ping, traceroute, iperf) to see if problem is path-wide; then isolate host using ethtool/ip -s/ss; packet captures confirm if loss is before/after host stack (drops in kernel vs retransmits on wire). Involve infra team when host evidence points to switch/port errors or upstream shaping.

MediumTechnical

144 practiced

Explain how connection saturation to a remote dependency (e.g., an API or database) can lead to cascading failures. Give an example architecture and outline short-term mitigations you would employ immediately to stop the cascade.

MediumTechnical

76 practiced

A backend service shows high lock contention leading to poor scaling. Explain three strategies to reduce contention at the code and architecture level and the performance trade-offs of each.

Sample Answer

High-level: contention happens when many threads serialize on the same synchronization point. Three practical strategies are:

1) Finer-grained locking / lock striping- What: Replace one global lock with multiple locks protecting disjoint subsets of data (e.g., per-bucket, per-resource).- Why: Reduces the probability threads need the same lock so true parallelism increases.- Trade-offs: More complex code, risk of deadlocks if multiple locks are acquired in varying orders, higher memory cost for many locks. Increased bookkeeping and harder reasoning about invariants.

2) Optimistic concurrency / lock-free primitives- What: Use atomic operations (CAS), Concurrent data structures (ConcurrentHashMap), or versioned updates (optimistic locking, retry on conflict).- Why: Readers and low-conflict writers avoid blocking; throughput improves when conflicts are rare.- Trade-offs: If conflicts are frequent, retries waste CPU and can starve progress. Lock-free code is harder to implement and reason about; may increase memory (copies) and latency variance.

3) Partitioning / remove shared state- What: Shard state by key (consistent hashing), run per-shard workers, or adopt event-driven architectures (CQRS, actor model) to localize mutable state.- Why: Completely eliminates cross-shard locks and scales horizontally; simpler single-threaded logic per shard.- Trade-offs: Increased operational complexity (routing, rebalancing), potential for skewed hot shards, cross-shard coordination becomes more expensive (distributed transactions or compensation logic).

Small example (CAS counter in Java):

java

AtomicInteger counter = new AtomicInteger(0);
void inc() { counter.incrementAndGet(); } // lock-free, low latency

When to pick:- Use finer-grained locks for quick wins in monoliths.- Use lock-free/optimistic where conflicts are infrequent and low-latency matters.- Use partitioning when you can redesign ownership and need large-scale horizontal throughput.

Monitor (contention metrics, tail latency) and measure after each change — sometimes combining approaches (sharding + lock-free per shard) yields best results.

Unlock Full Question Bank

Get access to hundreds of Scalability & Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.