Scalability Fundamentals Questions

Core concepts and back of the envelope estimation techniques for junior to intermediate engineers. This includes converting business requirements into technical metrics such as requests per second, data volume, and bandwidth; understanding when a single machine is insufficient and when to move to distributed systems; basic vertical versus horizontal scaling trade offs; basic sharding, replication, and caching patterns; monitoring signals to track capacity such as CPU trends and disk usage growth; and considerations for backup and recovery times and maintenance windows. Emphasis is on foundational calculations and practical guidelines for when and how to scale.

MediumSystem Design

39 practiced

Design a maintenance and upgrade strategy for stateful services to achieve near-zero downtime. Include rolling upgrades, blue-green, canary releases, data migrations, backup windows, and how to coordinate schema changes that are backward-compatible across versions.

Sample Answer

Requirements:- Minimize user-visible downtime for stateful services during maintenance and upgrades.- Support safe schema/data migrations, ability to roll forward/back, observability, and automated rollback.

High-level strategy:1. Preparation- Define compatibility guarantees and API/version matrix.- Add feature flags and versioned APIs.- Create automated integration tests exercising old+new versions and migrations.- Ensure backups/snapshots and tested restore playbooks; schedule low-impact backup windows.

2. Deployment patterns- Rolling upgrades: upgrade nodes one at a time, respect leader/follower roles; drain traffic, wait for health checks and replication catch-up before proceeding.- Canary releases: deploy to a small subset (1-5%) with metrics gating (errors, latency, DB errors). Promote gradually.- Blue-green for risky changes: provision new environment, migrate data or run dual-read logic, switch router/traffic once green passes smoke tests.

3. Data migrations & schema changes- Use expand-then-contract pattern: a) Expand: add new columns/tables/indexes nullable or with defaults; deploy code that writes both old and new formats (dual-write) and reads old-first then fallback to new. b) Backfill: run background jobs to populate new fields; monitor. c) Switch read-path: update services to read new fields. d) Contract: once all clients use new schema, remove old fields in a final safe deployment.- Use feature flags to gate migration-dependent behavior.- For stateful clusters, use versioned protobufs/serializers and tolerate unknown fields.

4. Backups & maintenance windows- Schedule full backups outside peak; use incremental backups for RPO improvement.- For long migrations, accept maintenance window for expensive operations (reindexing) but keep reads served via read-replicas or blue-green.- Test restore regularly.

5. Coordination & orchestration- Orchestrate via CI/CD pipelines supporting canary gating and automated rollbacks.- Use service mesh/load balancer to shift traffic.- Maintain a runbook: checklist, stakeholders, metrics to watch, rollback steps.- Communicate windows to stakeholders and enable on-call escalation.

6. Observability & rollback criteria- Monitor error rate, latency, DB replication lag, queue sizes, business metrics.- Define hard gates: if breached, stop rollout and roll back canary nodes or switch back blue environment.

Trade-offs:- Dual-write increases complexity and temporary storage cost.- Blue-green requires double capacity.This approach prioritizes safety: make migrations reversible, test compatibility, and only switch traffic when metrics validate stability.

EasyTechnical

33 practiced

Your product manager expects 500,000 monthly active users (MAU). On average each user has 3 sessions/day and each session triggers 12 API calls. Average single-request latency is 200ms. Calculate: (a) average RPS, (b) peak RPS assuming 10% of daily requests occur in the busiest hour, and (c) approximate concurrent requests. Show assumptions and steps and mention any additional factors you might include in a production estimate.

MediumSystem Design

37 practiced

You need to design sharding for a user-owned-content service where some users (celebrities) are extremely hot and cause uneven load. Propose a sharding and mitigation strategy that minimizes cross-shard operations, handles hot keys, and supports rebalancing. Include approaches like sub-sharding, dedicated shards, consistent hashing, and caching.

Sample Answer

Requirements & constraints:- User-owned content: reads dominated by followers of a user; writes from owner.- Minimize cross-shard ops (fan-out, joins), handle extreme hot keys (celebrities), support online rebalancing with minimal downtime.

High-level approach:1. Primary sharding by owner_id using consistent-hashing ring (or range/partitioning depending on query patterns) so a user’s content lives on one shard — minimizes cross-shard reads for owner-centric queries.2. Hot-key mitigation: - Dedicated shards: detect very hot owners (metrics: QPS, CPU, I/O). Move them to dedicated shard(s) sized for their load. - Sub-sharding (fanout partitioning): for a hot owner, split their content across N sub-shards by hashing (owner_id, content_id) mod N. Maintain a small routing map for that owner to avoid global lookup. - Rate-limiting and write queues for bursts to smooth spikes.3. Caching & materialized aggregates: - Use CDN / edge cache for public read-heavy feeds. - Per-owner cached timeline (precomputed fan-out or fan-in) with TTL; invalidate/refresh on writes. - For hot owners, keep hot-data in an in-memory cache (Redis cluster) colocated with their shard to avoid DB hits.4. Routing layer: - A lightweight service (router) that maps owner_id -> shard or owner -> sub-shard map. Cache mappings in router with TTL and fallback to consistent-hash lookup.5. Rebalancing: - Use virtual nodes in consistent hashing to move ranges gradually. - For moving heavy owners, do live migration: dual-write to old+new shard, backfill new shard, switch routing when sync complete; then quiesce old shard. - Autoscaling: monitor metrics and spin up dedicated shards or increase sub-shard count for specific owners.6. Minimize cross-shard ops: - Keep follower lists sharded by follower_id to avoid scans; for fan-out reads, precompute or use read fan-in (merge sorted iterators) with paging. - Avoid joins across shards by denormalizing (store owner metadata with content or in fast cache).

Trade-offs:- Dedicated shards and sub-sharding add routing complexity and storage duplication (for caches), but keep latency low for hot keys.- Precomputing feed reduces read cost but increases write complexity; choose hybrid (precompute for heavy users, compute-on-read for light users).

Example flow:- Read timeline for follower: router -> determine which owner shards to query -> for hot owner, hit Redis cache for precomputed feed; aggregate remaining owners’ feeds by merging paged results.

This design balances low cross-shard work, isolates hotspots, and supports safe rebalancing.

HardSystem Design

38 practiced

Your microservices platform experiences cascading failures when a downstream service slows. Design resilience patterns (bulkheads, circuit breakers, timeouts, retries, fallback handlers) and describe how you'd instrument and test these protections to avoid whole-system outages. Give an example of how to partition services into bulkheads.

Sample Answer

Requirements & constraints:- Prevent one slow/failed downstream from taking down the platform; keep latency/p99 and availability SLOs; allow graceful degradation.- Support heterogeneous services, scale independently, and keep operational visibility.

High-level resilience patterns- Timeouts: fail fast at each RPC boundary (e.g., client-side timeout = slightly less than caller's deadline — typical starting values: 200–500ms for sync calls, longer for batch/async).- Circuit Breakers: trip after N failures or error rate threshold (e.g., 5 failures or >50% errors in a 30s window), move to OPEN for T_open (30s) then HALF-OPEN with controlled probes.- Retries: idempotent-only, limited attempts (1–2 retries), exponential backoff with jitter (e.g., base 100ms, max 1s), and only for transient errors (5xx, timeouts).- Bulkheads: isolate resources (thread pools, connection pools, process/container quotas) so one downstream doesn’t exhaust threads or sockets for others.- Fallback Handlers: graceful degradation — cached responses, stale-but-consistent data, reduced feature set, or user-facing message. Prefer quick deterministic fallbacks.

How to partition into bulkheads (example)- By downstream dependency: give each external service its own client thread pool + connection pool + circuit-breaker instance. E.g., PaymentServiceClient uses pool size 50, CatalogServiceClient uses pool 100.- By functionality/criticality: isolate critical flows (auth, payments) from non-critical ones (analytics, recommendations) into separate service instances or Kubernetes deployments with separate node/pod quotas.- By customer tier: high-paying customers get dedicated pools/instances to protect SLAs.- By tenancy boundary: multi-tenant systems can isolate noisy tenants into separate pools or namespaces.

Instrumentation & observability- Metrics per client: request rate, latency histogram (p50/p95/p99), error count, circuit-breaker state, pool utilization, retry counts, fallback rates.- Tracing: distributed traces (OpenTelemetry) to see where latency accumulates; tag traces with downstream IDs and bulkhead pool.- Logs & events: structured logs for breaker-open, fallback-executed, retry-exhausted.- Dashboards & alerts: SLO-based alerts (e.g., p99 latency > SLO, fallback rate > threshold), pager escalation on cascading failure patterns.- Correlate incidents with topology: map service dependencies.

Testing & verification- Unit & contract tests: verify client logic—timeouts, retry/backoff, correct error classification, fallback behavior.- Integration & staging: inject deterministic delays and errors via a fault-injection layer or proxy (e.g., Envoy fault filter).- Load tests: ramp upstream traffic while downstream is degraded to ensure bulkheads limit resource usage and that circuit-breakers and fallbacks engage.- Chaos & chaos experiments: use Chaos Monkey/Chaos Toolkit to kill or slow downstreams in production canaries; run game days to validate runbooks.- Canary rollouts: gradual deployment with real traffic throttled to detect regressions.- Observability tests: synthetic probes and alerting playbooks; verify alerts fire and on-call can remediate.

Trade-offs & rationale- Strong isolation reduces blast radius but increases resource usage and configuration complexity.- Aggressive timeouts and fallbacks improve stability but may reduce feature richness; tune via metrics and canary experiments.- Circuit-breaker thresholds and retry policies must be tuned per downstream behavior and SLOs.

SummaryImplement client libraries with standardized timeout/retry/circuit-breaker behavior, enforce per-dependency bulkheads (thread/connection/pod), provide deterministic fallbacks, and build rich telemetry. Validate with unit/contract tests, fault-injection in staging, load testing under degraded conditions, and periodic chaos exercises to ensure the platform survives slow or failed downstreams without cascading outages.

EasyTechnical

33 practiced

Explain vertical scaling (scale-up) versus horizontal scaling (scale-out). For each approach, list three advantages and three disadvantages. Then describe concrete signals or thresholds (CPU/memory/disk/latency) you would monitor to decide that vertical scaling is no longer sufficient and horizontal scaling is needed for a service.

Sample Answer

Vertical scaling (scale-up) means increasing capacity of a single node — bigger CPU, more RAM, faster disks. Horizontal scaling (scale-out) means adding more nodes and distributing load across them (load balancers, sharding, replicas).

Vertical scaling — Advantages:- Simple: usually fewer architecture changes, often only resizing VM/instance.- Lower operational complexity: single system to monitor and back up.- Strong single-threaded performance: easier for workloads that don’t parallelize.

Vertical scaling — Disadvantages:- Upper limits: hardware has finite max capacity.- Single point of failure: node crash affects whole service unless replicated.- Cost inefficiency at scale: high-end machines have diminishing returns and higher per-unit cost.

Horizontal scaling — Advantages:- Infinite-ish scalability: add nodes to grow capacity linearly.- Fault tolerance: node failures have limited blast radius; easier to maintain availability.- Cost-effective at scale: commodity servers or containers scale out cheaply.

Horizontal scaling — Disadvantages:- Complexity: requires distributed systems concerns (consensus, partitioning, eventual consistency).- Operational overhead: orchestration, service discovery, load balancing, and cross-node testing.- Data consistency/performance trade-offs: cross-node coordination can add latency.

Signals/thresholds indicating vertical scaling is insufficient (examples):- Sustained CPU utilization > 70–80% across peak windows despite optimizations.- Memory pressure: >75% used with frequent GC/ OOM events or large swap usage.- Disk I/O saturation: high IOPS or disk utilization >70% with growing queue length and high latency.- Application latency: p95/p99 latency increasing and SLO breaches during load even after faster CPU/memory.- Load patterns: increased concurrent connections beyond single-node capacity or inability to parallelize work.

When you see sustained resource saturation plus rising tail latencies and scaling costs rising nonlinearly, move to horizontal scaling (replication, sharding, async work queues) and redesign stateful bottlenecks for distribution.

Unlock Full Question Bank

Get access to hundreds of Scalability Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.