Infrastructure Scaling and Capacity Planning Questions

Operational and infrastructure level planning to ensure systems meet current demand and projected growth. Topics include forecasting demand headroom planning and three to five year capacity roadmaps; autoscaling policies and metrics driven scaling using central processing unit memory and custom application metrics; load testing benchmarking and performance validation methodologies; cost modeling and right sizing in cloud environments and trade offs between managed services and self hosted solutions; designing non disruptive upgrade and migration strategies; multi region and availability zone deployment strategies and implications for data placement and latency; instrumentation and observability for capacity metrics; and mapping business growth projections into infrastructure acquisition and scaling decisions. Candidates should demonstrate how to translate requirements into capacity plans and how to validate assumptions with experiments and measurements.

HardTechnical

55 practiced

How would you scale a metadata store (like Hive Metastore or AWS Glue) that struggles with millions of partitions and thousands of small files per day? Discuss caching strategies, denormalizing frequently-accessed metadata, metastore sharding or federating, partition pruning and compaction strategies, and introducing a lightweight index service to reduce planning time and load.

Sample Answer

Start by clarifying SLAs (planning latency, metadata staleness tolerance, write throughput). Then apply layered changes so the metastore can scale without a risky big-bang migration.

Architecture & high-level approach:- Split responsibilities: authoritative metastore (truth) + read-optimized caches/indexes for planning.

Caching strategies:- Multi-layer cache: local in-executor cache (per Spark worker) + shared distributed cache (Redis/ElastiCache or memcached) for hot table/partition metadata.- Cache keys: table schema, partition list prefixes, per-partition file manifests, statistics.- Use TTLs + versioning/invalidations on metadata writes (event-driven invalidation via Kafka/notifications) to keep caches fresh while avoiding full reloads.- Use LRU and size-based eviction; keep small, frequently-used items (schema, recent partitions) always cached.

Denormalize frequently-accessed metadata:- Store precomputed denormalized objects: partition ranges, partition column min/max, file counts, approximate size, column statistics, and small-file markers.- Maintain compact manifests per partition (list of file paths + sizes) to avoid repeated expensive listing of object storage.- Keep both coarse-grained summaries (for pruning) and fine-grained manifests (for planning).

Metastore sharding / federation:- Horizontal sharding by namespace/table/creation time (e.g., shard per team, per database, or per year/month for time-partitioned data).- Federate: keep lightweight catalog service that routes queries to the correct shard; allow per-shard HA and independent scaling.- Use consistent hashing or directory-based mapping; avoid cross-shard transactions—use eventual consistency for partition additions.

Partition pruning & layout:- Enforce good partitioning strategy (avoid high-cardinality columns; consider bucketing or hive-style multi-level partitions).- Surface partition predicate pushdown to the planner using denormalized min/max stats to prune whole partition ranges without listing files.- Maintain bloom filters or column sketches per partition to speed selective queries.

Compaction & small-files mitigation:- Continuous small-file compaction pipeline: micro-batch job that consolidates small files into larger ones based on thresholds (size, file count, age).- Use file format with good append/merge semantics (Parquet/ORC) and optimize file sizes for the read workload.- For streaming ingestion, write to staging buckets then periodically compact and atomically swap manifests.

Lightweight index service for planning:- Build an index service (RocksDB, DynamoDB, Elasticsearch) storing: partition keys, per-partition file manifest pointer, stats, and file-level bloom filters.- The planner queries the index (low-latency) to get candidate partitions and file lists; only then access object store.- Make index updates asynchronous on writes but strongly consistent for new partitions if required by SLA (or provide version tokens).

Operational considerations & trade-offs:- Monitor cache hit rates, planner latency, metastore QPS, and number/size of file listings.- Accept eventual consistency for faster reads; provide transactional path for jobs that need strict consistency.- Cost vs complexity: managed caches and DynamoDB cost more but reduce planning time significantly.- Roll out incrementally: enable caching + index for busiest tables first, then apply sharding and compaction.

Expected outcomes:- Dramatic reduction in planning time (orders of magnitude) by avoiding cold listings and full partition scans, reduced metastore load, and fewer small files leading to faster reads and lower cloud egress/listing costs.

MediumSystem Design

64 practiced

How would you scale stateful services in Kubernetes—such as a metadata store used by pipelines—while ensuring data durability and minimizing downtime? Discuss PVC resizing strategies, StatefulSet vs Deployment choices, PodDisruptionBudgets, node autoscaler interactions, storage class capabilities (volume expansion, snapshots), and recovery strategies for autoscaler-driven events.

Sample Answer

Requirements & constraints:- Durable metadata store (strong consistency), minimal downtime during scale, autoscaler may add/remove nodes, storage must support expansion/snapshots, RTO low.

High-level approach:- Use StatefulSet for single-writer or leader-based metadata stores (Zookeeper, etcd, Postgres with Patroni). Deploy with stable network IDs, ordered startup/termination and Headless Service.- Only use Deployment for stateless or clustered stores that manage membership independently.

PVC & storage class:- Choose a StorageClass that supports volume expansion and snapshots (CSI driver with online expansion and CreateVolumeSnapshot). Use ReadWriteOnce for dedicated volumes or RWX if the store supports shared filesystem.- Pre-provision larger PVCs when possible; rely on online volume expansion for growth.

PVC resizing strategies:- Monitor capacity/IO via alerts and autoscale PVCs proactively. Use Kubernetes VolumeExpansion to expand PVs with minimal pod disruption. For systems that require filesystem resize inside pod, coordinate a rolling restart with PodDisruptionBudget (PDB).

Availability controls:- Define PodDisruptionBudgets to guarantee at least N replicas remain Available (e.g., minAvailable = replicas-1 for quorum stores).- Use PodAntiAffinity to spread replicas across nodes/zones.

Node autoscaler interactions & recovery:- Configure Cluster Autoscaler with safe eviction settings; mark pods with critical priorityClass to avoid eviction during scaling-in when necessary.- Use Pod Disruption Budgets and Pod Priority to influence scale-in; for pods that must not be evicted, use nodeGroup size and maintenance windows instead.- Implement lifecycle hooks: preStop to gracefully demote leader, leader election that transfers before termination, and readiness probes to prevent traffic during resize.

Snapshots & backups:- Regular automated snapshots via CSI snapshots and off-cluster backups (S3/GCS) to ensure recovery. Store metadata/state and WALs.

Recovery strategies:- On autoscaler-driven termination or node failure, rely on StatefulSet restore policy + PVC retain/delete as appropriate. Use snapshots to restore a new PVC and StatefulSet ordinal mapping. For leader-based stores, automate leader re-election and WAL replay.- Test chaos scenarios and recovery runbooks (restore time targets, steps).

Trade-offs:- StatefulSet + RWX can complicate HA; expanding PVs online depends on underlying CSI. More aggressive autoscaler policies reduce cost but increase risk—favor conservative scale-in for critical metadata stores.

This combines storage features, Kubernetes primitives, and operational playbooks to preserve durability and minimize downtime.

MediumTechnical

74 practiced

You must change a widely-used streaming table schema (add a nullable column and change a field type). Describe a safe, zero-downtime migration approach covering schema registry updates, producer and consumer compatibility, staged rollout, backward/forward compatibility rules, testing strategy (integration and replay), and verification checks to ensure downstream consumers are unaffected.

Sample Answer

Approach (high level): do this in two safe, zero-downtime phases—(A) add the new nullable column, and (B) move/change the field type via an additive migration pattern—so consumers are never forced to read an incompatible wire format.

1) Schema-registry and compatibility rules- Set Schema Registry compatibility to BACKWARD (or BACKWARD_TRANSITIVE) during additive change. This allows new writers to add nullable fields while old readers still succeed.- For a type change that is not a compatible Avro promotion (e.g., string -> int), do NOT modify the existing field in-place. Instead add a new field (e.g., foo_v2) with the desired type and mark the old field as deprecated. That preserves backward/forward compatibility.

2) Producer changes and staged rollout- Phase 1 — Add nullable field: - Register new schema that adds nullable "new_field" with default null. - Deploy producers in stages (canary -> 10% -> 50% -> 100%) that start writing the new_field when available. Producers continue to write old fields unchanged.- Phase 2 — Migrate type: - Update producers to populate both old_field (legacy type) and new_field (new type) for a transitional window (dual-write). Use feature-flag or config to enable dual-write. - After consumers are migrated, stop writing old_field and eventually remove it.

3) Consumer compatibility & rollout- Consumers must be resilient: when reading, prefer new_field if present, else fallback to old_field (with conversion logic).- Rollout consumer updates progressively: run canaries that validate both code paths. Update downstream ETL jobs to read new_field for logic and keep fallback until old_field is retired.

4) Testing strategy- Unit & contract tests: update Avro schemas and use Schema Registry's compatibility checks in CI.- Integration tests: spin up local Kafka + Schema Registry, produce sample data with both old and new schemas; run consumers to assert no exceptions and correct field resolution.- Replay testing: capture a copy of production topic (mirror) and replay into a staging cluster where migrated consumers run against real traffic.- Property tests: fuzz messages that contain only old, only new, both, and malformed combos.

5) Verification and monitoring- Automated checks: schema validation metrics, consumer deserialization error rate, consumer lag, POP (percentage of messages containing new_field).- Data correctness: row counts, key-field invariants, unit tests comparing UDF outputs between old and new consumers on mirrored data.- Alerting: set thresholds for error rate or data divergence to auto-roll back producers.- End-to-end verification: run downstream analytic queries and compare aggregates between pre-change and post-change windows.

6) Rollback & cleanup- If issues arise, flip producers back to old schema (they can still write compatible records), stop canary traffic, and investigate.- Only remove deprecated old_field after 2+ weeks of zero reads and production verification.

Why this is safe- Adds fields only (phase 1) keeps wire format compatible.- Dual-write and fallback ensure no consumer breaks during transition.- Staged rollout and mirrored replay exercise real traffic and surface issues early.- Schema Registry compatibility and CI gate keep incompatible changes from being deployed accidentally.

EasyTechnical

74 practiced

Given an SLA that requires p95 query latency < 500ms and monthly availability 99.9%, outline a practical SLO and alerting strategy to detect capacity problems before SLA breaches. Include which metrics to monitor (and at what aggregation), warning vs critical threshold levels, alert routing, and immediate mitigation steps the on-call should take.

Sample Answer

SLO proposal- Service SLO: p95 query latency < 500 ms and success rate ≥ 99.9% on a rolling 30-day window for production query endpoints used by analytics.- Operational SLOs (early-warning): rolling 30-day p95 < 400 ms; success rate ≥ 99.95% (tighter so we detect drift early).

Metrics to monitor (aggregation & windows)- p95 query latency: 1m (raw), 5m (alert eval), 1h (trend/automation). Alert on 5m and 1h windows.- Error / success rate (HTTP 5xx, RPC failures): 1m and 1h rolling.- QPS (queries/sec) and request size: 1m.- Backlog / queue length (ingest queues, Kafka consumer lag): 1m.- Resource metrics for query nodes: CPU, memory, disk IO, network — 1m granularity, 5m aggregated.- Thread/connection pools and GC pause times: 1m.- Downstream storage contention (e.g., DB latency, read replica lag): 1m/5m.

Warning vs Critical thresholds (example)- p95 latency (5m): Warning ≥ 350 ms, Critical ≥ 450 ms (SLA at 500 ms).- Success rate (5m): Warning drop > 0.2% from baseline, Critical drop ≥ 0.5% absolute or success < 99.5%.- QPS: Warning if sudden +50% vs baseline (5m), Critical if +100% or sustained increase causing latency.- Consumer lag: Warning if lag grows > 10k msgs or increases steadily for 10m; Critical if lag doubles in 5m or backlog threatens retention.- Resource exhaustion: Warning at 70% CPU/memory, Critical at 90%+ or OOMs.

Alert routing & behavior- Warning alerts: routed to data-engineering on-call via chat (Slack) + non-paging email. Include recent graphs, runbook link, correlation hints (deployments, traffic change).- Critical alerts: pager to on-call (phone/SMS) and escalate to senior data eng + platform/SRE if not acknowledged in 5 min. Auto-attach runbook, last deploy, top-10 offending queries, consumer lag snapshot.- Suppression: suppress transient spikes (<2m) and dedupe correlated alerts. Use alert grouping by service/region.

Immediate on-call mitigation steps (runbook)1. Confirm: check p95, error rate, QPS, recent deploys, region, and top slow queries.2. Rollback/hold deploys if a deployment coincides with regression.3. Throttle or rate-limit heavy analytical queries; enable maintenance mode for ad-hoc dashboards if supported.4. Scale horizontally: increase query-worker instances or read-replica count; increase spark/executor parallelism for batch-induced pressure.5. Relieve backpressure: increase consumer parallelism, restart stuck consumers, or move partitions to healthy consumers.6. Clear transient resource issues: restart unhealthy nodes, increase instance size (vertical) as short-term fix.7. If storage DB is slow, switch to read-replica, enable caching layer, or route queries to fallback faster store.8. Communicate: post incident status to stakeholders and open incident ticket with actions and ETA.

Post-incident- Conduct blameless postmortem, adjust SLO thresholds if false positives, add synthetic load tests, and add capacity autoscaling rules (scale on p95 latency + consumer lag) to prevent recurrence.

MediumSystem Design

76 practiced

You must validate cluster sizing for Spark jobs that process 10 TB of daily data and should complete within 2 hours. Design a load-testing and benchmarking plan listing representative job variants, dataset sampling strategy, executor/core/memory configs to test, shuffle/partition tuning to vary, and key metrics to collect (shuffle I/O, GC, CPU, task skew). Explain how to convert test results into a production cluster sizing and scheduling plan that supports concurrency.

Sample Answer

Requirements & constraints:- Process 10 TB/day and finish within 2 hours (batch window).- Support N concurrent jobs (assume concurrency target provided; I'll show how to compute).- Use Spark cluster sizing based on measured per-GB throughput and shuffle behavior.

Load‑testing & benchmarking plan (stepwise):1. Representative job variants- CPU-bound map-heavy ETL (few shuffles): e.g., parsing + enrichments.- Shuffle-heavy aggregation (groupBy, joins, sort).- I/O-bound read/write heavy (large parquet/ORC scans + writes).- Mixed pipeline with UDFs and wide transformations.For each variant test both full-job and streaming-window equivalents if applicable.

2. Dataset sampling strategy- Use production schema and distribution. Create datasets at 1%, 5%, 10%, 25%, 50% of 10 TB (100GB, 500GB, 1TB, 2.5TB, 5TB).- Ensure sampling preserves key cardinalities and skew (use stratified sampling on keys).- For shuffle-sensitive tests, inject realistic hot keys to measure skew.

3. Executor/core/memory configs to test- Executors per worker: test small (2–4 executors/node) vs large (1 executor/node).- Cores per executor: 1, 2, 4, 5 (avoid >5-6 to limit task contention).- Executor memory: test combinations to keep overhead (spark.yarn.executor.memoryOverhead) ~7-10%.- Try container sizes: e.g., 8c/64GB, 16c/128GB, 4c/32GB variants.- Test dynamic allocation on/off.

4. Shuffle / partition tuning to vary- numPartitions: test default, 2x, 4x, 8x of (inputSize / targetPartitionSize) where targetPartitionSize ∈ {128MB, 256MB, 512MB}.- spark.shuffle.compress true/false, spark.shuffle.spill.compress.- Shuffle service on/off, reduceParallelism (spark.sql.shuffle.partitions).- Broadcast thresholds (spark.sql.autoBroadcastJoinThreshold).

5. Key metrics to collect (per test)- Job wall-clock time and stage timings.- Shuffle I/O: bytes read/write, shuffle file counts, shuffle spill to disk.- Task metrics: task duration distribution, stddev, max (to detect skew).- CPU: core utilization per node (user/system), per-executor CPU.- Memory: executor JVM heap usage, GC pause times (young/old), memory pressure.- Disk I/O and network throughput (MB/s), latency.- GC: frequency and pause distribution.- Spark executor lost/retry counts, task serialization/deserialization times.- Cost metrics: vCPU-hours, storage and egress.

How to run tests:- For each dataset scale and job variant, run matrix of executor/core/memory configs and partitioning settings. Start with smaller scale to validate correctness then scale up.- Repeat runs 3x to account for variability; capture median and 95th percentile.

Converting results to production sizing & scheduling1. Compute stable per-GB throughput:- From best stable run for each job variant, compute throughput = input_bytes_processed / job_time (GB/hr or GB/s).- Use median of repeated runs; prefer runs with low GC/spill.

2. Per-job resource needs:- Record effective cores used and memory per executor in the best run.- Derive per-GB CPU-hours and memory-hours.

3. Sizing for 10 TB in 2 hours:- Required aggregate throughput = 10,240 GB / 2 hr = 5,120 GB/hr.- If best run throughput per cluster unit (e.g., per node type, or per core) is T GB/hr, needed nodes = ceil(5120 / T).- Alternatively compute cores required: needed_cores = ceil( (5120 GB/hr) / (GB/hr per core) ).

4. Support concurrency:- If C concurrent jobs required, multiply resource needs by concurrency factor.- Use bin-packing: separate heavy-shuffle jobs on nodes with fast disks and high network; CPU-bound jobs on compute-heavy nodes.- Reserve headroom: add safety factor (1.2–1.5) to account for variance, GC spikes, and future growth.- Plan for preemption: schedule non-critical jobs in lower-priority pools or off-peak windows.

5. SLA & Autoscaling rules- Define rules: min nodes = baseline to ensure liveliness; scale-up when pending tasks > X or average scheduler delay > Y; scale-down only after sustained idle period.- Use pod/node labels to isolate noisy neighbors (e.g., shuffle-heavy separate node pool).

Verification & monitoring- Deploy sized cluster with node pools matching tested configs; run full 10 TB dry-run.- Monitor metrics in real-time, compare to benchmarks; tweak partitions/GC configs.- Implement alerts on shuffle spill, GC > threshold, task skew > threshold.

Deliverables- Benchmark matrix (variant x dataset scale x config) with metrics and recommendations.- Sizing worksheet: GB/hr per core/node, nodes required, concurrency plan, safety factor.- Automation scripts for reproducible tests and dashboards for ongoing monitoring.

This approach converts repeatable micro-benchmarks into concrete node/core/memory counts and scheduling policies while accounting for shuffle behavior, skew, and variance.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Scaling and Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.