Technical Depth and Systems Thinking Questions

Assessment of deep technical expertise in one or more domains combined with systems level thinking and architectural judgment. Candidates should be able to explain the design and inner workings of complex systems or components they have built, describe why particular technologies and patterns were chosen, and evaluate trade offs across performance, cost, reliability, maintainability, and security. Interviewers will probe system boundaries and cascading effects, failure modes and mitigation strategies, scalability approaches, observability and monitoring choices, deployment and operational considerations such as continuous integration and continuous delivery, and how design decisions affected business outcomes. At senior levels, expect discussion of technical leadership, ownership of architectural direction, mentoring decisions, and evidence of measurable impact or value delivered. The scope includes both generic system design reasoning and concrete walkthroughs of one or two high complexity projects where the candidate can tie technical choices to impact metrics.

MediumSystem Design

41 practiced

Design a safe rollback mechanism for ML model deployments so that if a new model fails, traffic can be shifted back without losing incoming requests or violating SLOs. Include discussion of blue/green, canary, traffic-splitting, model artifact immutability, and coordination with feature/transform changes.

Sample Answer

Requirements:- Zero/low request loss and meet latency/error SLOs during rollback.- Safe rapid rollback (seconds–minutes).- Support feature/transform coupling and artifact immutability.- Observability and automated rollback triggers.

High-level architecture:Client → Load Balancer/Service Mesh (Envoy/Istio) → Traffic Splitter → Model Serving Pods (versions immutable) → Metrics + Logging → Rollback Controller (automation + runbook)

Key components & responsibilities:1. Immutable model artifacts: store models with content-addressable IDs (hash), signed manifests, and versioned schema for inputs/outputs.2. Blue/Green deployment: deploy new model set (green) alongside production (blue) with separate backing stores and transforms. Validate readiness before routing any traffic.3. Canary / traffic-splitting: start with 0.5–5% traffic to canary via service mesh weighted routing; gradually increase upon metric thresholds.4. Observability: collect latency, error rate, prediction quality (if ground truth available), feature-distribution drift, and request-level tracing.5. Rollback Controller: enforces automated rollback when SLOs or anomaly detectors trigger (e.g., error rate > X% or latency > Y ms or data drift). Also supports manual rollback with one-click.6. Coordination for features/transforms: bundle model + transform code in the same immutable release artifact; use compatibility checks (schema, unit tests) and migration plan if transforms change shape.7. Sticky sessions & in-flight requests: use connection draining and per-request routing in mesh to ensure new requests go to chosen version while in-flight complete. For streaming/long-running inferences, use tokens to track assigned model; allow both versions to serve until requests complete.8. Data consistency & stateful components: version feature stores and ensure backward-compatible reads; if feature extraction changes require migration, keep old transform available for blue until migration final.

Traffic shift and rollback sequence:- Deploy green (immutable) behind feature-compatibility gate.- Start canary: route small %; run shadowing of 100% for offline scoring if needed.- Monitor metrics; if within thresholds, progressively increase.- On trigger: immediately reduce weight to 0 and route to stable; initiate drain period to finish in-flight requests; mark canary as failed and roll back via routing policy; optionally garbage-collect after manual confirmation.

Trade-offs:- Blue/green uses more resources but simplifies quick rollback.- Canary reduces risk and cost but needs sophisticated metrics and gating.- Bundling transforms increases deployment size but prevents mismatches.

Best practices:- Automate safe rollback with predefined SLOs and circuit-breaker rules.- Keep artifacts immutable and signed.- Maintain backward-compatible input/output schemas or explicit migration strategies.- Run regular chaos tests and simulate rollbacks to validate runbooks.

HardSystem Design

39 practiced

Design an end-to-end CI/CD pipeline for ML models that includes: deterministic training runs, dataset and feature validation, model evaluation gates (performance/regression), automated promotion through staging, model artifact signing, and safe deployment with rollback. Name key stages, tools you would integrate, and failure-handling behavior.

Sample Answer

Requirements & constraints: reproducible (deterministic) training, dataset/feature validation, automated evaluation/regression gates, promotion through staging, artifact signing, safe deploy + rollback, observability.

Pipeline stages (linear + event-driven checks):1. Source & Data Lock- Tools: Git (model code), DVC or Delta Lake for dataset versioning, Feature store (Feast).- Behavior/failure: commit triggers pipeline; if data hash in DVC not found -> fail early and notify.

2. Reproducible Build- Tools: Dockerfile + conda/pip lock, Bazel or MLflow Projects, container registry.- Determinism: pinned seeds, fixed library versions, checksum of environment.- Failure: build test fails -> stop and open PR.

3. Data & Feature Validation- Tools: Great Expectations / TFDV; schema checks, drift detection, null/constraint checks.- Behavior: run on training slice and production sample; if schema drift or invalid features -> block and create incident ticket.

4. Deterministic Training & Logging- Tools: Kubeflow Pipelines / Airflow / Tekton to orchestrate, Spark/PyTorch/TF; W&B or MLflow for experiment tracking.- Ensure determinism: seed control, single-node for critical reproductions, save git commit, data hashes, env hash into run metadata.- Failure: training job retries with exponential backoff; if reproducibility mismatch (checksum != expected) -> fail and alert.

5. Model Evaluation & Gates- Tools: MLflow Model Registry + custom evaluation service.- Tests: unit tests, performance thresholds, statistical tests (p-value), regression tests against baseline, fairness and explainability checks.- Gate behavior: automated pass -> promote to staging; if marginal fail -> notify humans for manual review; if fail -> rollback to previous good candidate and mark PR failed.

6. Artifact Signing & Registry- Tools: cosign / GPG to sign container + model artifact; store artifacts in secured model registry (MLflow, S3 with immutability).- Failure: unsigned artifacts not allowed; signing failure -> halt promotion.

7. Staging Deployment & Canary- Tools: Kubernetes (namespace staging), Seldon/KServe or KFServing for serving; Istio/Linkerd or API Gateway for traffic routing.- Strategy: shadow & canary (start at 1% traffic -> ramp). Automated health checks, latency/SLA monitors.- Failure: auto-rollback triggers if error rate or degradation beyond thresholds; circuit-breaker to divert to baseline.

8. Approval & Promotion to Prod- Tools: CD system (ArgoCD / Flux) with GitOps; manual approval step optional (SLAs determine).- Behavior: signed artifact + pass gating tests required. Promotion writes immutable tag in GitOps repo.

9. Production Deployment & Monitoring- Tools: Prometheus, Grafana, OpenTelemetry, SLO/Alerting, A/B testing platform.- Safety: continuous shadowing for new data slices, feature-monitoring (Feast + drift detectors).- Failure: automated rollback to last signed artifact; incident automation pages on-call, attach run metadata for root-cause.

Cross-cutting concerns:- Metadata lineage: use MLflow + data catalog (Amundsen) to trace experiment -> data -> model.- Access & security: RBAC, secrets in Vault, signed artifacts only.- Observability: centralized logs (ELK), metric-driven alarms, periodic retraining pipelines.- Cost/Scalability: use spot instances for training with checkpointing; autoscale serving.

Failure-handling behavior summary:- Fail-fast on data/schema/build errors, notify stakeholders.- Automated retries for transient infra errors.- Gate-based blocking on evaluation failures; human-in-loop for borderline cases.- Signed-artifact policy prevents unsigned deployments.- Canary + metric-based automatic rollback for production health regressions.- Post-mortem and automated rollback analysis with linked run metadata for faster fixes.

HardTechnical

42 practiced

Explain trade-offs when using eventual consistency to synchronize model metadata (e.g., config flags, enabled features) across microservices in a multi-region system. How do you handle stale config reads, coordinate updates safely, and roll forward corrective changes when inconsistent states are observed?

Sample Answer

Trade-offs- Availability & latency vs correctness: eventual consistency gives low-latency, highly available reads across regions but allows stale reads. For ML metadata (feature gates, model-version pointers) this can cause inconsistent inference behavior.- Complexity vs reliability: stronger protocols (consensus, synchronous replication) reduce staleness but add latency, coupling, and cross-region outages risk.- Risk profile: categorize metadata—safety-critical (kill-switch, data-privacy toggles) demands stronger consistency; experimental flags can tolerate eventual semantics.

Handling stale reads- Versioned metadata: attach monotonic version/timestamp and source-region ID so services can detect staleness.- TTL + refresh-on-miss: cache with short TTL and background refresh; on read, if version < expected, trigger async refresh or fallback.- Serve-time validation: for critical changes, make the client require a minimum version (or check a control plane) before applying behavior.- Read-repair & shadow reads: occasionally read from primary or compare control-plane snapshot to detect divergence.

Coordinating updates safely- Use compare-and-swap / optimistic concurrency with version numbers to avoid lost updates.- For cross-region authoritative updates, designate write master per key or use a consensus layer (e.g., Paxos/Raft/Cloud Spanner) for critical keys.- Use transactional outbox + event streaming to publish change events reliably; include idempotency keys.- Staged rollouts: canary by region, percentage rollout, and health checks that gate further propagation.

Rolling forward corrective changes- Observability: log config versions in inference logs; alert on mixed-version traffic patterns or model metric regressions.- Automated remediation: if inconsistency detected, push corrective authoritative update with higher version; clients should accept higher-versioned updates even if older cached.- Forced reconciliation: periodic reconciliation job that compares central desired state vs regional state and repairs via retries/backoff.- Safe defaults and circuit breakers: if inconsistent or unknown, fall back to conservative model or disable risky features.- Auditing & manual kill-switch: ability to globally enforce a safety config via synchronous path when automated remediation is too slow.

Example policy (ML context)- All safety-critical flags: global leader + consensus write, synchronous confirmation before global rollout.- Model experiments: publish via event stream, use short TTL caches and allow divergence for up to X seconds, but require inference logs to record config version for downstream attribution and rollback.

Why this works- Balances latency and safety by categorizing metadata, uses versioning and reliable event delivery to converge, and provides observable, automated correction paths so model behavior remains auditable and recoverable.

MediumSystem Design

33 practiced

Design an inference serving architecture that supports 10,000 requests per second, a 500 MB model artifact, and a p95 latency target of 50ms. Include discussion of batching (static vs dynamic), model sharding/replication, hardware choices (CPU/GPU), network protocol (gRPC/HTTP2), autoscaling strategy, and how you'd estimate the number of replicas required.

Sample Answer

Requirements & constraints:- 10,000 RPS sustained, p95 latency <= 50 ms, model artifact size 500 MB.- Workloads: synchronous low-latency prediction requests.- Environment: Kubernetes (typical for MLE role).

High-level architecture:- Load balancer → Ingress (gRPC/HTTP2) → Front-end API pods (request validation, auth, routing) → Inference service pods (model replica container) → Metrics & autoscaler.- Observability: Prometheus, tracing, per-replica latency and queue metrics.- Deployment: Each replica loads the 500 MB model into process (and into GPU memory if using GPUs). Use sidecar for batching/queueing (e.g., NVIDIA Triton or custom dynamic-batcher).

Batching: static vs dynamic- Static batching: fixed batch size N. Pros: predictable throughput and utilization. Cons: added latency if not enough concurrent requests; poor for bursty or low concurrency.- Dynamic batching (recommended): aggregate requests up to max_batch_size or latency_budget (e.g., 8–16 ms) before executing. This maximizes GPU throughput while bounding added batching delay to meet p95 SLAs. Implement per-replica queue with backpressure and per-request deadline. Use frameworks like Triton or TorchServe dynamic batching.

Model sharding vs replication- 500 MB fits on common GPUs (e.g., 16–32 GB), so prefer replication over sharding for simplicity and lower latency.- Sharding only if model > GPU memory or each request needs different parts; sharding introduces cross-GPU communication and increases latency.

Hardware choices (CPU vs GPU)- If model is a deep net with large matrix ops -> GPU recommended for throughput per latency. Use inference-optimized GPUs (T4, A10, A100) with TensorRT/ONNX optimizations.- If model is small or heavily CPU-optimized (quantized), CPU instances may be cheaper for moderate throughput.- Hybrid: serve latency-critical requests on GPU pool; fallback/overflow on CPU pool.

Network protocol- Use gRPC over HTTP/2 with protobufs: lower overhead, streaming support, easier connection reuse and backpressure. Use TLS termination at LB. Keep requests small; binary payloads.

Autoscaling strategy- Use K8s HPA/VPA with custom metrics: - Primary signals: request queue length, p95 latency, GPU utilization or inference throughput. - Scale-up triggers: sustained queue length above threshold, p95 latency nearing 50 ms, GPU util > 70%. - Scale-down: graceful with cool-down, keep a warm pool (pre-warmed replicas) to avoid cold-start latency (model load ~seconds). - Use predictive/step-scaling for expected traffic patterns and KEDA for event-driven bursts.- Warm pool: maintain N idle replicas per zone to ensure quick capacity.

Estimating number of replicas (example calculation)- Measure per-device max throughput under batching while keeping p95 <= 50 ms. Two components of latency: - processing_time(batch): pure compute for batch - batching_delay <= latency_budget (we must keep total p95 <= 50ms)

Example empirical profiling:- On inference-optimized GPU with dynamic batching: - Batch size 16 → throughput 4,000 inf/s per GPU, processing latency ~8–12 ms - Batch size 8 → throughput 2,400 inf/s per GPU, processing latency ~6–8 ms- If dynamic batching yields effective sustained throughput ≈ 2,500 req/s per GPU while keeping p95 < 50ms (including batching delay), required GPUs = ceil(10,000 / 2,500) = 4 replicas.- Conservative baseline if uncertain: characterize locally. If CPU only and single replica handles ~20 req/s with p95 50ms, need 10,000/20 = 500 CPU replicas (or larger machines).

Practical steps to estimate:1. Run microbenchmarks: measure p50/p95 latency and throughput across batch sizes on target hardware with production model (including deserialization and postprocessing).2. Choose max_batch_size and latency_budget that keep p95 ≤ 50ms.3. Compute per-replica sustainable RPS = measured throughput under those settings.4. Add safety margin (20–30%) and multi-AZ redundancy: replicas_needed = ceil( RPS_target / (per_replica_RPS * (1 - safety_margin)) ).

Other considerations and optimizations- Optimize model: quantization, kernel fusion, TensorRT/ONNX for faster inference and lower memory.- Co-location: place front-end and inference in same AZ; use node labels/taints to schedule GPU pods.- Admission control & prioritization: reject or queue low-priority requests under overload.- Caching results for idempotent requests to reduce load.- Model loading time: use warm pools or snapshot warmers to avoid cold-starts (500 MB load can take seconds).

Trade-offs- Larger batches increase throughput but risk violating p95 budget. Dynamic batching with tight latency budget balances utilization and latency.- GPUs reduce replica count and cost-per-inference if model compute-bound; CPUs simpler but may require many replicas and induce higher total cost/infra.

Summary- Use gRPC/HTTP2, dynamic batching with per-request latency budget, replicate model on GPUs (unless model > GPU memory). Empirically measure per-device throughput with chosen batch strategy, then scale replicas = ceil(10k / per_replica_rps) with a 20–30% safety margin, autoscale on queue length/GPU utilization and keep warm pool to meet p95 50ms.

HardTechnical

60 practiced

You must roll out a new model to 100 million users while ensuring fairness across demographic slices and avoiding amplification of existing biases. Propose a rollout strategy: which fairness metrics to compute, pre-launch checks, staged rollout plan across demographic slices, and mitigation techniques if unfairness is detected post-rollout.

Sample Answer

Requirements & constraints:- Serve 100M users with minimal disruption, preserve utility, and ensure no amplification of historical biases across protected slices (race, gender, age, location, income proxy).- Target: parity within acceptable bounds (predefined by stakeholders/regulator).

High-level approach:1. Fairness metrics to compute- Group metrics: demographic parity gap, equalized odds (TPR/FPR gaps), calibration-in-group (Brier score per slice).- Individual/group uplift: disparate impact ratio, average treatment effect by slice.- Performance: AUC, accuracy, precision/recall per slice.- Robustness: confidence distribution shifts and error severity (harm-weighted error).

2. Pre-launch checks- Data audit: label quality by slice, sampling bias, missingness patterns.- Counterfactual simulations: synthetic reweighting to estimate potential amplification.- Offline fairness testing: compute metrics above on validation and holdout slices; stress tests on subgroups with low prevalence.- Shadow deployment: run model in parallel, log decisions and predicted probabilities; compare to incumbent on historical traffic.

3. Staged rollout plan- Canary (0.1%): internal users + enriched samples of vulnerable slices; real-time monitoring of fairness metrics and latency.- Incremental bursts (1% → 5% → 20% → 50% → 100%): at each stage require metric gates: no increase in harm-weighted error > X%, cap on TPR/FPR gaps (predefined), no disparate impact ratio below threshold.- Stratified sampling: ensure each stage oversamples underrepresented slices so signals emerge early.- A/B tests with uplift analysis: measure both aggregate utility and slice-level effects; use sequential testing with early-stopping rules.

4. Post-rollout mitigation if unfairness detected- Quick roll-back to previous model or isolate affected cohort routing.- Apply calibrated post-processing: threshold adjustments per slice to equalize TPR/FPR or minimize disparate impact.- Reweighting/recalibration: online importance weighting or score calibration per slice.- Retrain with fairness-aware objectives: add constraints (e.g., equalized odds) or adversarial debiasing; incorporate corrected labels if audit found label bias.- Human-in-the-loop escalation: route flagged high-risk decisions to human review and collect feedback for retraining.- Governance: incident report, root-cause analysis, update SLAs and monitoring dashboards.

Operational considerations- Define thresholds and SLAs with legal/product/stakeholders before rollout.- Automate continuous monitoring dashboards, alerts for metric drift, and automated throttles that pause rollout if gates fail.- Logging and privacy: log sufficient features for debugging while preserving privacy and compliance (PII minimization, secure access).

This plan balances rapid deployment with strong monitoring, slice-aware staging, and multiple mitigation levers to prevent and correct amplification of biases.

Unlock Full Question Bank

Get access to hundreds of Technical Depth and Systems Thinking interview questions and detailed answers.

Join thousands of developers preparing for their dream job.