Scaling Systems and Teams Questions

Covers both technical and organizational strategies for growing capacity, capability, and throughput. On the technical side this includes designing and evolving system architecture to handle increased traffic and data, performance tuning, partitioning and sharding, caching, capacity planning, observability and monitoring, automation, and managing technical debt and trade offs. On the organizational side this includes growing engineering headcount, hiring and onboarding practices, structuring teams and layers of ownership, splitting teams, introducing platform or shared services, improving engineering processes and effectiveness, mentoring and capability building, and aligning metrics and incentives. Candidates should be able to discuss concrete examples, metrics used to measure success, trade offs considered, timelines, coordination between product and infrastructure, and lessons learned.

EasyTechnical

52 practiced

You must capacity-plan GPUs for an online sequence-to-sequence model. Each GPU can serve ~20 concurrent inferences with average GPU inference latency of 50ms (including batching). Your workload target is 500 QPS with p95 latency target 200ms. Show your calculations for required GPUs, explain assumptions (batching efficiency, warm pools, headroom), and list other factors that might change your estimate.

Sample Answer

Approach and core calc:- Each GPU can run ~20 concurrent inferences; average latency per inference = 50 ms = 0.05 s.- Per‑lane throughput = 1 / 0.05 s = 20 RPS. With 20 concurrent lanes → GPU throughput ≈ 20 * 20 = 400 RPS (QPS).

Raw capacity required:- Target = 500 QPS.- GPUs (raw) = 500 / 400 = 1.25 → round up → 2 GPUs.

Assumptions and headroom:- We should not run GPUs at 100% steady utilization. Assume target sustained utilization 60–75% to protect p95 tail, account for jitter, and leave room for autoscaling decisions.- Using 70% target utilization: effective per‑GPU = 400 * 0.7 = 280 QPS.- GPUs needed = 500 / 280 ≈ 1.79 → round up → 2 GPUs, but because 1.79 close to 2, operational safety suggests at least 3 when factoring p95 and bursts.

p95 latency target (200 ms) considerations:- Average 50 ms < 200 ms, but p95 can be much higher under bursty load or larger batches. To keep p95 < 200 ms, maintain spare capacity so batching doesn't inflate queueing delay. That argues for 2 GPUs minimum and 3 GPUs recommended (2 active + 1 warm/standby) to serve spikes without queueing.

Practical recommendation:- Start with 2 GPUs for baseline, set autoscaling to add a 3rd GPU when sustained QPS > ~350 per active GPU (monitor queue lengths and p95), and keep a warm pool (1 pre‑warmed replica) to avoid cold startup latency.

Other factors that change the estimate:- Batching efficiency and batch size: larger batches improve throughput but raise per‑request latency and tail.- Arrival pattern (burstiness, diurnal peaks) and queueing effects.- Cold starts (model loading) and warm pool size.- Pre/post‑processing CPU or I/O bottlenecks (may reduce effective GPU throughput).- Model size (memory) limiting max concurrent batch size.- Network latency and client RTT contributions to p95.- Mixed precision/optimized kernels or multi‑model packing can increase throughput.- Failover/redundancy, maintenance windows, and SLA requirements (may require N+1).- Variability between GPUs, tenant isolation, and cost constraints.

Summary: Raw math gives 2 GPUs, but operational safety and p95 constraints make 3 GPUs (2 active + 1 warm/standby or autoscaled) a practical starting point; tune based on observed batching latency, p95, and arrival patterns.

EasyTechnical

50 practiced

At a high level, explain data parallelism versus model parallelism for training deep neural networks. Give example scenarios where each is appropriate (including hybrid approaches), and list the operational trade-offs (communication overhead, batch-size limits, memory usage, framework support).

Sample Answer

Data parallelism vs model parallelism — high-level:

Data parallelism:- Partition the training dataset across workers; each worker holds a full copy of the model and processes a different mini‑batch. Gradients are aggregated (e.g., all-reduce) and model parameters are synchronized.- Appropriate when the model fits a single device’s memory but you want to scale throughput (more GPUs → larger effective batch size and faster epoch time). Example: training ResNet on ImageNet across 8 GPUs.

Model parallelism:- Partition the model across devices; different layers or tensor slices live on different devices and a single batch flows through the pipeline. Useful when the model is too large to fit on one device.- Appropriate for very large language models or huge vision transformers that exceed single‑GPU memory. Example: sharding transformer layers across GPUs.

Hybrid approaches:- Combine data + model parallelism (e.g., tensor/model sharding inside data‑parallel groups, or pipeline parallelism across micro‑batches). This is common for billion-parameter models (Megatron-LM, DeepSpeed).

Operational trade-offs:- Communication overhead: Data-parallel all-reduce cost grows with model size; model-parallel requires frequent activations/gradients transfers between layers/devices. Hybrid complicates patterns.- Batch-size limits: Data-parallel scales batch size linearly with workers (may hit generalization limits); model-parallel keeps batch small but requires more fine-grained synchronization.- Memory usage: Data-parallel duplicates model per device (higher total memory usage); model-parallel splits model so per-device memory can be lower, but activation memory and communication buffers increase.- Latency vs throughput: Pipeline/model parallelism can increase per-step latency (pipeline bubbles) but allows larger models; data-parallel optimizes throughput.- Framework support & complexity: Data-parallel is well supported (PyTorch DDP, Horovod). Model/hybrid needs advanced tooling (PyTorch RPC/torch.distributed.pipeline, DeepSpeed, FairScale) and careful engineering (load balancing, checkpointing, failure recovery).

When choosing: if model fits single GPU and you need speed → data-parallel. If model is too big → model or hybrid. Consider network bandwidth, memory, and acceptable batch-size/accuracy trade-offs.

MediumSystem Design

40 practiced

Design a canary deployment process for rolling out a new model version in production. Include traffic split strategy, metrics to evaluate (both infra and model-quality), automated rollback criteria, monitoring windows, and how you'd run statistical tests to detect regressions in behavior. Describe how to handle multi-tenant customers during canarying.

Sample Answer

Requirements & constraints:- Zero/low customer impact, fast detection of regressions, tenant isolation, automated rollback.

High-level flow:1. Bake model + integration tests in staging.2. Deploy new model to a canary cluster/environment.3. Gradually route traffic with automated checks; promote to prod when safe.

Traffic-split strategy:- Phase 0: 0% (smoke)- Phase 1: 1% — random users, run for short window- Phase 2: 5% — broaden to varied geography and load- Phase 3: 20% — longer observation- Phase 4: 50% → 100% if greenUse traffic bucketing by user-id hashing to ensure deterministic assignment and sticky sessions.

Metrics to evaluate- Infra: request latency p50/p95/p99, CPU/GPU utilization, memory, OOMs, error rate (5xx), throughput, queue depth.- Model-quality: task-specific metrics (e.g., accuracy/F1, BLEU/ROUGE, perplexity), calibration (confidence distribution), latency of inference, top-k correctness, false positive/negative rates, user engagement (CTR, retention), safety metrics (toxicity, hallucination rate).- Business KPIs: conversion, revenue per user as applicable.

Monitoring windows & cadence- Short window checks: 5–15 min for infra (latency spikes, errors).- Medium window: 1–6 hours for model-quality metrics stable under sampling.- Long window: 24–72 hours for business impact and rare failure modes.

Automated rollback criteria- Threshold-triggered: any infra metric exceeding thresholds (e.g., p95 latency +30% or error rate > baseline + 0.5%) → immediate rollback.- Statistical degrade: model metric drop with statistical significance (see tests) beyond allowable delta (e.g., accuracy drop >1% with p < 0.01) → pause/rollback.- Safety triggers: toxicity/hallucination above threshold → immediate rollback.Rollback should be automated for hard infra failures; for model-quality, automated pause and alert to engineers for human-in-the-loop if borderline.

Statistical testing for regressions- Use A/B hypothesis testing between baseline and canary cohorts with pre-defined metrics.- For continuous metrics (latency, confidence): use t-test or bootstrap CIs; prefer non-parametric (Mann-Whitney) if non-normal.- For proportions (error rate, accuracy): use chi-square or proportion z-test.- Control for multiple comparisons (Benjamini-Hochberg) when monitoring many metrics.- Use sequential testing (alpha spending, e.g., O’Brien-Fleming) or Bayesian sequential monitoring to allow early stopping without inflating false positive.- Require both statistical significance and minimum effect size (practical difference) before acting.

Handling multi-tenant customers- Tenant-aware canarying: group tenants into tiers (pilot, standard, enterprise). Start canaries on low-risk / opt-in tenants first.- Per-tenant isolation: route specific tenant IDs to canary; track per-tenant metrics to catch tenant-specific regressions.- Allow tenant opt-in/opt-out and feature flags to control canary exposure.- For shared resources, ensure resource quotas per tenant so one tenant's canary load cannot affect others.- For high-SLA tenants, require separate approval and longer observation windows.

Operational considerations- Logging & trace sampling for canary traffic flagged for deeper diagnostics.- Dashboards with canary vs baseline overlays and autogenerated alerts.- Runbook for manual rollback, A/B deep-dive, and postmortem.- Periodically re-run canary with randomized seeds and data shifts to detect data drift regressions.

This process balances automation for safety with statistical rigor and tenant-aware controls to reduce risk while enabling rapid model iteration.

MediumTechnical

46 practiced

Compare synchronous (blocking) inference vs asynchronous (non-blocking / queued) inference pipelines. For an ML product with mixed workloads (real-time chat, batched offline scoring), when would you choose each approach? Discuss advantages, failure modes, SLAs, and how to transition a system from sync to async without disrupting clients.

Sample Answer

Synchronous (blocking) inference: client request waits for model response in the same call (low tail-latency goal). Asynchronous (queued/non-blocking) inference: request is accepted quickly, placed into a queue or job system; response delivered later (push, webhook, polling, or stored result).

When to choose each- Real-time chat / interactive UX: prefer synchronous to meet sub-100–300ms latency SLAs and immediate feedback. Use lightweight models, optimized serving (GPU/CPU pinning, batching within the server), and autoscaling for spikes.- Batched offline scoring, heavy transforms, or low-priority background work: prefer asynchronous to maximize throughput and resource efficiency (large batches on GPUs, scheduling during off-peak), tolerate minutes-to-hours latency.

Advantages- Sync: simple API/contract, predictable for clients, good for tight SLAs and simple error handling.- Async: higher throughput, better resource utilization (large batches), graceful degradation under load, easier to apply different QoS (priorities).

Failure modes & mitigations- Sync: head-of-line blocking, queue saturation causing high tail latency or dropped requests. Mitigate with request timeouts, circuit breakers, autoscaling, per-client rate limits, and backpressure.- Async: queue growth, task loss, out-of-order or stale results, harder troubleshooting. Mitigate with durable queues, retry policies, idempotency keys, TTLs, and monotonic timestamps.

SLA considerations- Define latency percentile targets (p50/p95/p99). For async, specify acceptable end-to-end completion windows and visibility (e.g., accepted-at timestamp, estimated completion).- Provide differentiated SLAs: “interactive” (sync, 200ms p95) vs “bulk” (async, 1–24h).

Transitioning from sync to async without disrupting clients1. Clarify contract: publish new async API and SLA docs; maintain old sync endpoint during migration.2. Non-breaking API: add an endpoint or header to opt into async rather than changing sync semantics.3. Dual-write mode: accept requests synchronously but also enqueue for async processing; or accept and respond with immediate acknowledgement plus job ID.4. Progressive rollout: pilot with a subset of clients or percentage (feature flag) and monitor metrics (latency, queue length, error rates).5. Backwards compatibility: support polling/webhook for results; provide client libraries that abstract polling and retry.6. Observability & compensations: add tracing, job-level logs, alerting on queue backlog; implement fallbacks (fast approximations) if backlog threatens SLA.7. Deprecation plan: after clients adopt async, deprecate sync with ample notice.

Trade-offs summary- Use sync for strict-interaction latency and simple contracts; use async for throughput-heavy, expensive, or non-urgent work. Combine both: hybrid systems that route by priority and provide graceful degradation (e.g., sync fast-path to a smaller model, async full model).

HardSystem Design

48 practiced

Design a privacy-aware logging and observability approach for model inference that minimizes PII exposure but retains enough signal for debugging and monitoring. Include log redaction strategies, tokenization/anonymization, differential privacy considerations, retention policies, and how to enable safe replay for debugging while complying with regulations.

Sample Answer

Requirements:- Minimize PII exposure while preserving signals for debugging/monitoring.- Allow safe replay for debugging under strict controls.- Meet GDPR/CCPA and sector-specific rules (health/finance).- Low latency, scalable, and auditable.

High-level architecture:Client → Inference Proxy (redaction + policy) → Model Server → Observability Pipeline (ingest, store, analytics) → Access/Replay Service (secure, audited)

Core components & responsibilities:1. Inference Proxy (edge): apply deterministic redaction rules (regex, NLP NER) to strip obvious PII (SSNs, emails, phones). Replace with stable tokens (tokenization) and attach token mapping IDs stored in a separate encrypted vault with limited access.2. Tokenization/Anonymization: use irreversible hashing with per-tenant salt for analytics; reversible token vault only for authorized debug requests with multi-party approval and time-limited decryption.3. Logging/Observability Pipeline: ingest logs via secure channel; enrich with non-PII metadata (model version, latency, input shape, aggregated category tags).4. Differential Privacy (DP): for aggregated telemetry (metrics, usage patterns, failure clusters) apply DP mechanisms (Laplace or Gaussian) before dashboards to prevent reconstruction.5. Retention & Access Controls: tiered retention — high-fidelity logs (with tokens) kept encrypted for short window (e.g., 7–30 days), aggregated/DP'ed summaries retained longer. Enforce RBAC, signed requests, MFA, and keep audit logs.6. Safe Replay Workflow: require signed ticket + justification; replay runs against isolated debugging environment; reversible token mapping accessible only via time-bound HSM-backed decryption and logged approvals; redact outputs before exporting.7. Compliance & Monitoring: automated policy tests, periodic privacy impact assessments, and logging of all decryption/replay actions for audits.

Data flow & trade-offs:- Redaction at proxy minimizes PII egress but risks losing debugging signal; tokenization preserves linkage without exposing raw PII.- Reversible vault increases risk surface; mitigate with HSM, least privilege, and short retention.- DP reduces fidelity for small cohorts; balance epsilon choice to preserve utility.

Scalability:- Stateless proxy with autoscaling; stream processing (Kafka + Flink) for real-time redaction/aggregation.- Store encrypted token vault in scalable KMS/HSM-backed service.

Key metrics to monitor:- % of fields redacted, decryption request rate, mean time to approve replay, privacy budget consumption.

This approach balances privacy, compliance, and observability by combining automated redaction, tokenization with guarded reversibility, DP for analytics, strict retention, and an auditable safe-replay process.

Unlock Full Question Bank

Get access to hundreds of Scaling Systems and Teams interview questions and detailed answers.

Join thousands of developers preparing for their dream job.