Strategic Technical Decision Making Questions
Focuses on high-level, organization-wide technical decisions that have multi-year consequences, not the operating mechanics of any single technology. Candidates should reason through concrete decision types such as: choosing or consolidating a technology platform, deciding whether to build in-house versus buy or adopt a vendor/open-source solution, committing to (or reversing) a major architecture or infrastructure direction, sequencing technical debt paydown against feature velocity, and standardizing tooling or platforms across multiple teams. Strong answers name the alternatives considered, the criteria used to evaluate them (cost, reversibility, team capability, time horizon, risk), how uncertainty was managed with incomplete information, and how the rationale was communicated to non-technical stakeholders and leadership to build buy-in. This topic is about the reasoning, trade-off framing, and communication behind a strategic technical bet, not a definitional quiz on any specific technology (e.g. this is not the place for 'define eventual consistency' or 'explain canary deployments' style questions; those belong to the underlying technology topics).
MediumSystem Design
42 practiced
Design an API gateway layer for ML services that handles model routing, A/B experimentation, authentication, throttling, and observability while adding minimal latency. Describe component responsibilities, how to route by version or experiment, and what telemetry to emit at the gateway.
Sample Answer
Requirements & constraints:- Low added latency (sub-10ms goal)- Support model routing by version and experiment (A/B, canary, rollout)- Auth, per-customer throttling, observability, minimal churn for ML teamsHigh-level architecture:Client → Edge LB → API Gateway (stateless, hot path) → Model Router → Authz Service (cached) → Experiment Service (decision & config cache) → Throttler → Model Serving (gRPC/HTTP) → Logging/Obs pipeline (async)Component responsibilities:- API Gateway (fast, in-process): validate schema, authenticate token (JWT/mTLS; verify signature with locally cached keys), perform request normalization, add trace id, consult local in-memory routing cache, enforce throttling tokens, forward enriched request.- Model Router: evaluates routing rules (version header, user attribute, experiment assignment). Runs locally with periodic sync of rules from Experiment Config Store.- Experiment Service (control plane): defines experiments, traffic split rules, sticky assignment logic, and exposes k/v configs. Persists evaluation metrics for experiment analysis.- Throttler: token-bucket per-customer and global rate limits. Use Redis/Lua for distributed rate limits when needed.- Authz Service (control plane): issues/rotates keys, maps tokens to tenant/quota.- Observability pipeline: async exporter (Kafka) to metrics/traces/logs store.Routing by version or experiment:- Priority order: explicit client-specified version > sticky experiment assignment > default model.- Assignment algorithm: deterministic hashing on stable key (user_id, device_id). For weighted splits, hash mod 100 vs thresholds. Sticky cookie/header ensures consistent user experience.- Canary/rollout: support rules like “X% of requests for tenant T to version v2 where latency improvement > threshold” driven by Experiment Service flags. Gateway decision is local (no network call) using cached flag + short TTL (e.g., 1s) to minimize latency.- Header-based overrides for debugging (admin header whitelist).Minimizing latency:- Keep decision logic in-process, use read-through caches for auth keys and experiment configs with background refresh.- Use binary protocols (gRPC) to model servers, keep connections warm (connection pools).- Async, batched logging to reduce blocking; sample traces under high load.Telemetry to emit at gateway (per-request labels: tenant, model_version, experiment_id, route_outcome):- Metrics (prometheus): - request_count{tenant,model_version,experiment,endpoint,status} - latency_ms_histogram{...} (client->gateway, gateway->model, end-to-end) - error_count{...} (4xx/5xx, auth failures, throttle rejections, model errors) - throttled_count{tenant} - routed_count_by_rule{rule_id}- Traces (distributed): trace id across gateway → model server; spans for gateway decision, auth, throttling, model inference.- Logs: structured request logs (id, tenant, model_version, experiment, input_size, resp_size, latency_breakdown). Keep PII out; redact sensitive fields.- Feature-distribution sketches/sample payloads (privacy aware): periodically emit sample of inputs per model for drift detection (store in separate telemetry pipeline with access controls).- Experiment metrics: per-experiment conversion/quality signals and model-specific latencies/error rates, aggregated and exported to Experiment Service.Example flow:1. Gateway receives request, validates JWT (local cache).2. Lookup routing config locally; hash user_id → assignment (e.g., experiment E: user -> v2).3. Check throttler token; if exceeded return 429 with X-RateLimit headers.4. Forward to model v2 via gRPC, add trace headers.5. Asynchronously push metrics and structured log to observability pipeline.Trade-offs:- Local caches add eventual consistency on config changes; mitigate with short TTL and push invalidation for urgent rollouts.- Heavy telemetry sampling reduces visibility; tune sampling and aggregate histograms for scale.This design gives ML teams fast, flexible routing and rich telemetry while keeping the gateway latency minimal and the control plane expressive for experiments and rollouts.
MediumTechnical
39 practiced
Describe how model CI/CD and registry interactions should be architected to support reproducibility, auditing, and compliance for production ML models. Focus on metadata and integration points for serving (e.g., immutable artifacts, dependency declarations), without specifying particular data pipeline implementations.
Sample Answer
Start with a clear principle: treat models as immutable, versioned artifacts with rich metadata and signed provenance; CI/CD and the model registry are the contract layers that enforce reproducibility, auditability, and compliance.Key components and metadata- Immutable artifact: model binary/container image plus checksum (SHA256) and storage URI. Once registered, artifact is immutable.- Provenance (who/what/when): training job id, git commit SHA, dataset snapshot identifier (or fingerprint), training code container image, hyperparameters, random seeds, compute environment hash.- Dependency declarations: exact package versions, OS/base image, framework version, GPU/CPU ABI, and a lockfile-like artifact (e.g., pip freeze or conda env YAML) referenced by hash.- Model contract/schema: input/output schemas, feature manifest, model signature (types, ranges), and expected latency/throughput SLOs.- Validation artifacts: unit/integration test results, evaluation metrics (train/val/test), fairness checks, explainability reports, and compliance attestations (manual approvals, automated checks).- Governance metadata: approval records, change requests, retention policy, and expiry/periodic review timestamps.CI/CD–Registry interactions (integration points)1. Build/Train → Register: CI job publishes the immutable artifact + metadata to the registry. The registry assigns a deterministic version and stores checksums and provenance. CI must sign the artifact (cryptographic signature or provenance attestation).2. Pre-deploy gating: CD queries registry metadata to enforce gates—automated tests, metric thresholds, security scans, and required approvals. Only artifacts meeting gates are promoted to staging/production.3. Deployment manifest: Serving reads a manifest from the registry that includes artifact URI, checksum, runtime dependency bundle, model signature, and deployment constraints (resource requests, canary percentage).4. Serve-time verification: On fetch, the serving system verifies checksum and signature, resolves declared dependencies (pinned images or sidecar envs), and enforces schema checks before activation.5. Observability link-back: Serving annotates telemetry with model version and registry id; logs and metrics are linked back to the registry for audit and root-cause.6. Rollback & lineage: Registry exposes provenance so CD can roll back to a previous immutable artifact; audit trails show who promoted what and when.Reproducibility & compliance practices- Hash everything: code, data fingerprints, environment lockfiles—store hashes in registry for bit-for-bit reproducibility.- Signed attestations: cryptographically sign artifacts and approval events.- Immutable promotion model: promotion (staging→prod) never mutates artifacts; it only changes registry state/metadata.- Periodic re-evaluation: scheduled checks that re-run critical validations against production artifacts (data drift, retrain triggers) and record outcomes.- Fine-grained audit logs: every registry action (register, promote, approve, deploy, revert) with user/service identity and timestamp.- Least-privilege automation: CI/CD agents operate with scoped credentials; human approvals are recorded with identity verification.Trade-offs and rationale- Storing full data snapshots increases reproducibility but has storage/privacy cost—use dataset fingerprints plus policies for long-term retention when necessary.- Strict immutability and pinned dependencies maximize reproducibility and auditability but require robust dependency management to avoid environment rot.This architecture makes the registry the source of truth for model artifacts and metadata; CI/CD enforces gates and records attestations; serving systems perform verification and link operational telemetry back to registry entries for full end-to-end reproducibility and auditable compliance.
MediumSystem Design
50 practiced
Design a multi-region model-serving architecture that keeps tail latency low for global users. Address traffic routing, model synchronization or deployment strategy, CDN/caching for static assets, cross-region failover, and how to limit cost and complexity while providing reasonable consistency for features.
Sample Answer
Requirements & constraints:- Low tail latency for global users, reasonable consistency for features, cost/complexity constrained, robust cross-region failover.High-level approach:- Active‑active multi‑region serving with geo-routing and regional model replicas, lightweight cross-region failover and eventual-consistent feature sync.Architecture components:1. Traffic routing- Global DNS/GSLB with latency-based routing + Anycast front door (Cloud CDN/Global LB). Route users to nearest healthy region.- Region-local load balancer -> autoscaling model-serving pods (K8s/ECS) behind mTLS API gateway. Use client affinity when short sessions & stateful batching needed.2. Model deployment & synchronization- CI/CD: model registry (artifact + metadata), containerized model images. Push images to global registry.- Deployment strategy: staged Canary → Blue/Green per region. Promote to all regions once canary metrics stable.- Artifact sync: push images to regional container registries; use object-store replication (cross-region replication) for large artifacts.3. Feature consistency & serving data- Serve features from a hybrid approach: - Read-mostly features: asynchronously replicate feature-store materialized views to each region (near-real-time, eventual consistency). - Strongly consistent features: route writes to a single primary region or central transactional store when necessary (use sparingly).- For per-request freshness, use TTLs and confidence metadata so model can degrade gracefully.4. CDN/caching- Use CDN for static assets (models manifests, client SDKs, embeddings if static).- Cache deterministic model outputs at edge/region for repeated identical requests (use hash key + short TTL).5. Cross-region failover & health- Health checks in GSLB; if region unhealthy, fail over to next-best region automatically.- Data: accept reduced consistency during failover; fallback policies for writes (queue & forward) and serve last-known-model/features.6. Cost & complexity controls- One canonical model image built once; lightweight regional replicas.- Use model compression (quantization/knowledge distillation) and adaptive batching to reduce infra cost.- Limit number of regions to coverage+latency curve analysis (e.g., 3–6 regions).- Automate canary/promote to avoid manual ops.Trade-offs:- Active-active with asynchronous feature replication trades strict consistency for latency and availability — acceptable for most ML features. For business-critical consistency, isolate small set of features with strong consistency (single-region writes or global DB with higher cost).Monitoring & SLOs:- Per-region latency p50/p95/p99, model accuracy drift, feature staleness metrics, failover time. Automated rollback on SLA or accuracy regressions.This design balances low tail latency via geo-routing and regional replicas, reasonable consistency through selective strong vs eventual consistency, and cost control through single build artifacts, compression, and limited regional footprint.
HardTechnical
74 practiced
You must recommend between building region-based specialized GPU inference clusters versus relying on CPU autoscaling with aggressive model quantization and distillation. Create a multi-year evaluation covering cost, latency, maintenance, scalability, cloud vendor dependencies, and ability to support new models.
Sample Answer
Clarify assumptions: target QPS per region (e.g., 500–5k), 50ms P99 latency target for real-time endpoints, model family (Transformer-based: 6B–70B params), 3–5 year horizon, traffic growth 30%/yr, cloud on-demand + reserved options, tolerance for occasional quality degradation (business rules).Year-by-year evaluation summary:Year 0–1 (MVP)- CPU + aggressive quantization/distillation: - Cost: lowest up-front; use autoscaling on existing infra. Example: distilled 6B → 500M params, INT8/FP16 inference reduces per‑request CPU cycles by ~4–10x. - Latency: moderate; median OK, P95–P99 risk for peak bursts unless overprovisioned. - Maintenance: high ML R&D to distill/quantize and maintain accuracy; CI/CD for model variants. - Scalability: good elastic scaling; autoscaler handles bursty traffic. - Vendor dependence: minimal, portable. - New models: slower — each new model needs retraining/distillation and evaluation.- Region GPU clusters: - Cost: high fixed infra/setup (GPU instances, regional clusters), but predictable per‑inference when mature. - Latency: best for large models and consistent low P99. - Maintenance: DevOps for cluster ops, driver/stack updates. - Scalability: harder for sudden spikes without spare capacity or elastic GPU pools. - Vendor dependence: stronger (GPU instance types, availability zones). - New models: fast to roll out (no distillation required).Year 2–3 (scale)- CPU path cost grows with QPS: autoscaling multiplies cost; larger models harder to quantize without quality loss. TCO may cross GPU breakeven for models > ~2B params at sustained high QPS.- GPU clusters amortize initial cost; with spot/preemptible and instance pooling, per‑inference cost drops. Latency remains superior. Maintenance ops stabilize; however, capacity planning and cross-region replication add complexity.Year 4–5 (maturity)- If traffic and model size grow as assumed, hybrid becomes optimal: GPUs in high-demand regions for large models/low-latency SLAs; CPU-quantized replicas for fallback, low-cost batch, and edge regions.- Trade-offs: GPUs lock you to vendor instance types and supply; CPUs give portability but require ongoing ML compression work and may hit accuracy ceilings.Recommendation and implementation plan:1. Start hybrid MVP: use CPU-quantized/distilled model for global low-cost baseline + small GPU pools in major regions (top 3) for high-concurrency and latest model variants.2. Metrics: per-request cost, P50/P95/P99 latency, model quality delta (BLEU/ROUGE/accuracy), SLA breach rate, deployment lead time for new models.3. Yearly review gates: - If sustained QPS > X and per-request CPU cost > 0.6× GPU per-request cost → expand GPU footprint. - If quantization causes >2% business-metric degradation → prioritize GPU serving for that workload.4. Operational needs: automated distillation pipelines, model canarying, cluster autoscaler for GPUs, multi‑region failover, reserved capacity for predictable traffic, spot for cost efficiency.5. Vendor-risk mitigation: design abstraction layer (model-serving API + containerized runtime) to allow switching instance families or hybrid cloud.Conclusion: For a 3–5 year horizon with growing model sizes and steady/high QPS, adopt a staged hybrid approach—start with CPU-optimized baseline and selectively expand region-based GPU clusters where latency and model fidelity justify the TCO.
HardSystem Design
46 practiced
Design a rollback and disaster recovery process for a fleet of stateful personalization model servers deployed across multiple regions. Ensure minimal data loss, consistent personalization state, and fast recovery. Address backup frequency, cross-region replicas, failover orchestration, and testing of the plan.
Sample Answer
Requirements & constraints:- Minimize data loss (RPO ≤ 5 min), fast recovery (RTO ≤ 15 min), consistent personalization state across regions, support rollback to prior model + state, multi-region availability, automated failover.High-level approach:- Treat each model server as stateful service composed of: model binary, local personalization state (user embeddings, feature accumulators), and a write-ahead log (WAL) for deltas. Use periodic snapshot + continuous WAL replication to cross-region storage and replicas. Orchestrate via Kubernetes with a custom Operator for safe failover/rollback.Core components & responsibilities:1. Local WAL (append-only): every state update is appended synchronously to local durable store (fast disk/ephemeral + replicated to node journal).2. Snapshotting: incremental snapshot every 5 minutes (tunable) to object storage (S3/GS) with versioned keys; full snapshot daily. This yields ~RPO 5m.3. Streaming replication: Push WAL entries in near-real-time to a cross-region Kafka or change-stream (CDC) topic for replay and to feed passive replicas in other regions (async with ack guarantees).4. Cross-region passive replicas: Deploy passive servers in each region that continuously replay WAL + snapshots to keep state warm (lag target <30s). Passive replicas are read-only until promoted.5. Metadata & control plane: Global config DB (strongly consistent store like Spanner/Cloud Spanner or a small RAFT cluster) stores active-version mapping, snapshot pointers, and last applied WAL offsets.6. Failover orchestration: Operator/Control-plane performs leader election + promotion. On region outage, promote passive replica with last-applied WAL offset >= control-plane checkpoint; shift traffic via global load balancer (DNS/Anycast/Traffic Manager) with health checks and drain time. Use canary traffic first (5-10%) then full cutover.7. Rollback: Model + state are versioned together. To rollback, mark desired snapshot+model version in control-plane, create a plan: (a) instantiate a fresh server from snapshot+WAL replay to consistent offset, (b) route canary traffic to it, (c) monitor metrics and promote.Consistency & correctness:- Ensure WAL entries are idempotent. Use monotonic per-user sequence numbers to apply in-order.- For strong consistency on critical keys, use per-user locks or use CRDTs if eventual convergence acceptable.- Use checkpointing: every snapshot stores last applied WAL offset; control-plane only promotes replicas whose offset ≥ target.Backup frequency & retention:- Incremental snapshots every 5 minutes; full daily snapshots.- WAL retention: keep at least 7 days; store in versioned, immutable S3 buckets with lifecycle rules (cold storage longer-term).- Retain cross-region replicas continuously replaying.Failover orchestration details:- Health discovery: liveness + state freshness (WAL-offset lag) metrics exported to monitoring (Prometheus).- Automated failover triggers: region-level loss of LB + replica lag > threshold OR operator manual trigger.- Stepwise failover: block writes to failed region (if split-brain risk), promote passive replica, switch DNS/traffic with short TTL, gradually increase traffic, verify consistency before full cutover.- For rollback: use blue-green model versioning, keep prior model+state snapshot available; perform shadow traffic tests; automated rollback if metrics degrade.Testing & validation:- Regular DR drills (quarterly) that simulate region outage and full rollback — measure RTO/RPO.- Chaos testing (chaos monkey) on replicas, network partitions, and WAL delays.- End-to-end integrity tests: synthetic users with deterministic sequences to verify replay produces same personalization outputs.- Canary and A/B tests for model+state changes; automated canary analysis comparing metrics.- Runbook + automated playbooks (Terraform/K8s + Operator CLI) for manual intervention with clear steps and rollback commands.Trade-offs:- Strong cross-region sync increases latency; mitigated by async WAL + warm passive replicas to keep latency low.- RPO vs cost: more frequent snapshots/WAL retention costs more; choose 1–5 min based on business value.- Choice of control-plane store: Spanner/Cloud Spanner for strong consistency vs etcd (smaller scale).Key failure modes & mitigations:- Split-brain: prevent by central control-plane quorum and disallow dual promotion.- Corrupted snapshot: keep checksums, multiple versions, and WAL-based rebuild.- WAL backpressure: rate-limit writes, fall back to local buffering with TTL and alerting.This plan ensures minimal data loss by frequent snapshots + WAL, consistent state via coordinated checkpoints and promotion rules, fast recovery by warm passive replicas and orchestrated traffic shifts, and confidence via automated testing and DR drills.
Unlock Full Question Bank
Get access to hundreds of Strategic Technical Decision Making interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.