Solution Approach & Modeling Strategy Questions

Techniques for approaching system design problems and architectural modeling in distributed systems, including problem framing, requirement elicitation, modeling abstractions (data flows, component boundaries, API interactions), trade-off analysis, and evaluation criteria for scalability, reliability, and maintainability.

MediumTechnical

60 practiced

Define SLIs and SLOs for an online product-recommendation API that must meet: 99th-percentile latency target of 150ms, availability 99.95% monthly, and maximum allowable model-quality regression of 0.2% relative to baseline. Explain how you'd measure SLIs, choose observation windows, set alert thresholds, and implement burn-rate-based incident responses.

Sample Answer

SLIs vs SLOs (brief)- SLI: a measurable signal of user experience (e.g., p99 latency, success-rate, model-AUC delta).- SLO: the target you commit to (e.g., p99 ≤150ms, monthly availability ≥99.95%, model-quality regression ≤0.2%).

Concrete definitions for the recommendation API- Latency SLI: p99 request latency measured at the API ingress (end-to-end server + model invocation), computed per-tenant and aggregated. SLO: p99 ≤ 150ms over a 30‑day rolling window.- Availability SLI: fraction of successful requests (HTTP 2xx) excluding planned maintenance and client errors (4xx). SLO: ≥99.95% per calendar month.- Model-quality SLI: relative change in chosen metric (e.g., offline NDCG@10 or business KPI like click-through-rate) versus baseline model on representative holdout. SLO: regression ≤0.2% over a 7‑day evaluation window.

How to measure- Collect traces and metrics at the edge (API gateway) with timestamps and request IDs. Use histograms for latency (hdr/histogram buckets) to compute p99 accurately.- Availability: compute successful_count / total_count after filtering 4xx and maintenance windows.- Model-quality: daily batch evaluation on a stable holdout; also compute online A/B metrics (control vs candidate) with bootstrapped confidence intervals.

Observation windows and rationale- Latency: 30-day rolling SLO for business commitment; shorter windows (5m, 1h, 24h) for alerts to catch regressions quickly.- Availability: monthly SLO with 1h and 6h alert windows for outages.- Model-quality: 7-day rolling for SLO to smooth noise; 24h for alerts if degradation is sustained.

Alert thresholds- Latency alerts: - Warning: p99 over last 1h > 120ms (80% of SLO) - Critical: p99 over last 5m > 150ms (breach) or sustained 1h >150ms- Availability alerts: - Warning: 1h error-rate causing projected monthly availability to dip below 99.98% - Critical: error-rate causing projected monthly availability <99.95% or spike >1% errors in 5m- Model-quality alerts: - Warning: observed regression >0.1% with p<0.05 over 24h - Critical: regression >0.2% over 48h or statistically significant drop in online KPI (95% CI excludes zero)

Burn-rate based incident response- Compute burn rate = (observed error budget consumed) / (fractional time elapsed). Error budget = 1 − SLO (e.g., availability budget = 0.0005 of time/month).- Define tiers: - Burn < 1 (normal): monitoring, no immediate escalation. - Burn 1–4 (high): page on-call, reduce noncritical deployments, start mitigation playbook. - Burn > 4 (critical): full incident response, roll back recent model/deployments, engage engineering + data-science, enable circuit breakers (fallback recommenders), throttle nonessential traffic.- For model-quality: treat regression as consuming a separate “quality” budget; high burn triggers model rollback to baseline, pause automated retraining, and run RCA.

Operational practices- Instrument every layer, correlate logs/traces/metrics.- Use canary & gradual rollout to avoid large burns.- Maintain playbooks: rollback, switch to cached recommendations, scale model-serving pods, and reroute to CPU/gpu pools.- Post-incident: blameless postmortem, update SLI measurement (if blind spots found), and adjust thresholds if needed.

This setup balances rapid detection (short windows & tight alerts) with business-level promises (30-day/monthly SLOs) and uses burn-rate to prioritize response and limit customer impact.

HardSystem Design

45 practiced

Design a scalable approach to provide per-request explanations (feature importance, counterfactuals, or saliency) for predictions from large models with minimal latency overhead. Consider options: on-the-fly explanation, approximate/surrogate explanations, cached explanations for repeated inputs, or asynchronous explanation retrieval. Address storage, consistency across model updates, and UX trade-offs.

Sample Answer

Requirements & constraints:- Per-request explanations (feature importances, counterfactuals, saliency) with minimal added latency (soft budget e.g. < 50–200ms).- High QPS, large model (GPU-backed), frequent model updates, auditability, and UX latency/accuracy trade-offs.

High-level architecture:- Inference path: Client → Inference Service (GPU) → Prediction returned immediately → Explanation service consulted in parallel.- Explanation service supports 4 modes: on-the-fly exact, fast surrogate, cached, and async-postback.

Components:1. Inference Service: returns prediction + request-id and metadata (model-version, seed).2. Explanation Router: - Fast path: look up Cache by fingerprint (input hash + model-version). If hit and TTL valid, return cached explanation instantly. - Surrogate path: if miss and latency budget tight, run a cheap surrogate/explainer (linear probe, distilled explanation model, feature-attribution approximator) on CPU; return approximate explanation. - Exact path (on-the-fly): if latency budget allows or user requested high-fidelity, schedule GPU-backed explainer (e.g., integrated gradients, SHAP via sampling) and wait synchronously. - Async path: for heavy explanations (counterfactuals) or post-hoc audit, enqueue job and notify client when ready via webhook or store in user UI.3. Explanation Store (immutable): stores explanations keyed by (fingerprint, model-version, explainer-config) with metadata, audit logs, and provenance. Use object storage + fast KV index (Redis/Mongo) for metadata lookup.4. Consistency & model updates: - Version explanations by model-version. Fingerprint includes model-version so cached explanations are invalidated automatically when model changes. - For small model patches, compute delta validation: sample inputs to check explanation drift; if within threshold, optionally reuse cached explanations (with flag).5. Storage strategy: - Keep high-fidelity explanations in cold storage (S3) and summaries/visualizations in fast store (Redis) for top-K, top-features. - Retain audit logs and provenance permanently or per governance retention policy.6. UX considerations: - Show immediate approximate explanation with “confidence/fidelity” score and option to view/call high-fidelity async result. - Allow users to request higher fidelity per-request (billing/latency trade-off). - Indicate model-version, explainer-type, and timestamp for transparency.7. Trade-offs: - Cache yields best latency but storage cost and risk of staleness; versioning mitigates. - Surrogates are fast but less faithful—must expose fidelity metrics and calibration. - On-the-fly exact is most faithful but costly and adds latency; restrict to premium/low-QPS paths. - Async supports heavy workloads and audits but reduces immediacy.

Operational concerns:- Monitoring: track cache hit-rate, explainer latency, fidelity drift between surrogate and exact explanations.- Cost control: tiered explainer SLAs, automatic eviction policies, and sampling-based exact explanation computation to limit compute.- Security & privacy: encrypt stored explanations, redact PII in stored artifacts.

Example flow:- User requests prediction; inference returns prediction + request-id.- Router checks cache (hit) → return cached explanation in <50ms.- If miss and user low-latency: run surrogate (20–80ms), return with fidelity score; enqueue exact explainer to populate cache and notify user when ready.

This design balances latency, fidelity, storage cost, and governance by offering multi-tiered explanation strategies, strong versioning for consistency, and transparent UX controls.

EasySystem Design

48 practiced

Describe and annotate the data flow for an online inference pipeline: client request → input validation → feature extraction/preprocessing → model inference → postprocessing → response. For each step, state responsibilities, potential failure modes, where to place boundaries (library vs service), and whether to implement synchronously or asynchronously. Highlight trade-offs in latency, debuggability, and deployability.

Sample Answer

Situation: An online inference pipeline processes client requests through six stages: input validation → feature extraction/preprocessing → model inference → postprocessing → response. Below is an annotated data flow with responsibilities, failure modes, service boundaries, and sync/async recommendations plus trade-offs.

1) Client request- Responsibility: Auth, rate-limiting, request formatting.- Failures: Auth fail, malformed payload, upstream latency.- Boundary: Service/API gateway.- Sync/Async: Synchronous accept; optionally async for long jobs.

2) Input validation- Responsibility: Schema checks, sanitize, basic semantic checks.- Failures: Invalid schema, malicious input.- Boundary: Library inside service for speed; small validations in gateway.- Sync/Async: Synchronous (reject fast).

3) Feature extraction / preprocessing- Responsibility: Tokenization, normalization, enrichment (lookup), caching.- Failures: Missing resources, slow external lookups, nondeterminism.- Boundary: Split: pure CPU ops as library; heavy enrichment (DB/feature store) as separate service.- Sync/Async: Synchronous for realtime features; asynchronous or batch-populated caches for expensive features.

4) Model inference- Responsibility: Run model, apply model-specific transforms, resource management (GPU/CPU).- Failures: OOM, model server crash, version mismatch, tail latency.- Boundary: Service (model server like TorchServe, KFServing, or hosted endpoint).- Sync/Async: Typically synchronous for low-latency; async for long-running/large models (return job id).

5) Postprocessing- Responsibility: Convert logits to business outputs, apply thresholds, calibrations, format results, safe-guards (toxicity filters).- Failures: Wrong thresholds, downstream logic bugs.- Boundary: Library or light service; keep business rules in service for easy updates.- Sync/Async: Synchronous.

6) Response- Responsibility: Serialize, apply compression, telemetry/logs.- Failures: Serialization errors, payload too large.- Boundary: Service/gateway.- Sync/Async: Synchronous return; for async flows return job status endpoint.

Trade-offs:- Latency: Libraries reduce RPC overhead → lower latency. External services add isolation but cost hops and tail latency.- Debuggability: Services provide observability, can version independently. Libraries make local debugging easier but entangle deploys.- Deployability: Service boundaries allow independent scaling and rollback (model servers separate), but increase operational complexity.- Fault tolerance: Services enable retries/circuit breakers and graceful degradation (fallback models or cached responses).Recommend: Keep deterministic, fast ops as libraries; isolate heavy/encrypted/stateful components (feature store, model server) as services. Use synchronous path for typical low-latency calls, and offer async job APIs for expensive or long-running inferences.

MediumSystem Design

51 practiced

Design a multi-region inference architecture for a conversational AI that must provide <100ms median latency globally, tolerate a single-region outage, and respect regional data residency requirements. Describe where models should be placed, how to handle state replication and caches, traffic routing (geo-DNS, anycast, edge), and how to perform safe failover and model rollout across regions.

Sample Answer

Requirements (clarify): <100ms median global latency, tolerate single-region outage, respect regional data-residency (certain user/session data must not leave region). High-level idea: place inference capacity in every region with local state and caches; route users to nearest region via geo-aware routing + anycast/edge; replicate non-residency-sensitive metadata asynchronously; provide controlled failover and safe model rollout with canary/shadow traffic.

Architecture summary:- Models: host identical model binaries (and small quantized/FP16 variants) in each region on GPU/TPU node pools (k8s/GKE/AKS/EKS with node groups). Keep a regional model registry/artifact store (or use global registry but pull policy enforces local hosting).- State and replication: - Session/PII: stored only in-region (encrypted at rest). Do not replicate cross-region unless legal/consent allows. - Non-PII metadata (logs, embeddings index shards, anonymized metrics): asynchronously replicated to other regions via event bus (Kafka/Cloud Pub/Sub) with eventual consistency. - For global features (cross-user personalization), use sharded design where data residency owners own shards; provide federated aggregation for global models.- Caches: - Per-region read-through cache (Redis/Memory) co-located with inference nodes for sub-ms lookup. - Warmed model caches on each node; model parameters served from local fast storage (NVMe) to avoid cold starts. - Avoid global cache for residency-sensitive keys.- Traffic routing: - Edge Anycast + regional ingress: use anycast for shortest-path to PoP, then geo-DNS or CDN routing to nearest healthy region. Edge performs TLS termination, basic auth/rate-limit and request pre-processing (tokenization/embedding if safe). - Health-aware geo-DNS: route users to another region when local region unhealthy. For residency-restricted users, geo-DNS must honor allowed failover map (e.g., EU->EU only).- Failover: - Active-active by default. Continuous regional health checks and latency SLIs. On single-region outage, traffic shifts using geo-DNS/route53 weighted failover + LB-level health checks. - For residency-restricted requests where cross-region failover is disallowed, return a graceful degraded response or emergency consent flow; optionally use cross-region proxy with explicit consent and auditable logs. - Circuit breakers and backpressure: enforce per-region concurrency limits, queue with retry/backoff.- Model rollout and safe deployment: - CI/CD: build image + model artifact, store in model registry with semantic versions and migration metadata. - Canary workflow: deploy new model to small percentage of nodes in each region (or to a single region first where business risk is low). Use shadow testing: mirror production traffic to new model without affecting user responses; compare metrics. - Gradual traffic ramp: weighted routing at LB/edge (5/20/50/100) per region. Automatically rollback on SLO regressions. - Blue-green or k8s rolling updates with draining: ensure in-flight requests complete; use model compatibility checks (schema, tokenization).- Performance tactics to achieve <100ms median: - Place inference in-region as close to user as possible (edge PoP or nearest AZ). - Use quantized/optimized model variants, batching tuned for latency (micro-batches), kernel-optimized runtimes (ONNX Runtime, TensorRT), and CPU fallback. - Warm pools / keep-alive instances to avoid cold starts. - Local caches for embeddings and recent responses.- Observability & testing: - End-to-end latency SLOs per region, per model-version. Distributed tracing, synthetic traffic from global probe agents. - Chaos testing: simulate region failures, network partitions, and validate failover and rollback automation.Trade-offs:- Full active-active per region increases cost (GPU footprint) but meets latency and resilience. Partial active-passive lowers cost but raises failover cold-start risk.- Strict residency prevents transparent failover; require product decisions for consent or degraded UX.

This design balances latency, resilience, and legal constraints by keeping sensitive state local, running inference capability in each region, using geo-aware routing with health/failover policies, and employing safe, observable model rollout practices.

HardTechnical

49 practiced

As a Staff AI Engineer leading the evaluation of modeling strategies for a new generative AI product, explain your process for aligning cross-functional stakeholders (product, safety, infra, legal), defining decision criteria (latency, cost, quality, safety), planning fast prototypes to validate trade-offs, and documenting the chosen direction and risks for execs. Include how you'd structure meetings, prototypes, and evidence for your recommendations.

Sample Answer

Situation: We're building a generative-AI feature and must pick a modeling strategy (on-prem large model, fine-tuned smaller model, or retrieval-augmented generation using API) with cross-functional buy-in.

Process / approach:1. Kickoff & alignment (week 0)- Invite product, safety, infra, legal, ops to a 60–90m workshop.- Output: clear success metrics (product OKRs), must-have constraints (latency < 300ms, P95), hard limits (data residency), and decision timeline.- Technique: decision rubric live-collaboration (weighted criteria).

2. Define decision criteria and weights- Propose criteria: quality (human eval, ROUGE/ROUGE-L/ BLEU where relevant, and task-specific accuracy), safety (toxicity rate, hallucination probability), latency (P50/P95), cost (per-1M requests), maintainability, and compliance.- Assign weights with stakeholders (e.g., safety 30%, quality 25%, latency 20%, cost 15%, infra/ops 10%).

3. Fast prototypes (2–4 weeks)- Parallel short spikes (1 week each) implemented by small teams: a) API-first: integrate a hosted LLM + prompt engineering + RAG. b) Fine-tuned mid-size model on constrained infra. c) Distilled on-device model (if applicable).- For each prototype define standard eval harness: synthetic benchmarks, 1000-sample human blind eval (labelers rate correctness, safety, helpfulness), latency tests under load, cost model (cloud + ops), and failure-mode tests (prompt injection, data-exfiltration simulation).- Keep experiments reproducible (notebooks, infra-as-code) and instrumented (metrics dashboards).

4. Meetings & cadence- Weekly 30–45m sync with reps for blockers and results.- Bi-weekly deep-dive to review prototype evidence (metrics + 10 example outputs per failure mode).- Rapid ad-hoc sessions for showstopping safety/legal findings.

5. Evidence package & recommendation for execs- One-page TL;DR: recommended option, top 3 trade-offs, short rationale.- 2–3 page appendix: decision rubric scores (spreadsheet), key metrics (quality/safety/latency/cost), representative examples (good, borderline, failure), risk register (likelihood, impact, mitigations), rollout plan (MVP -> staged rollout), and contingency paths.- Quantified ask: infra cost delta, hiring/ops needs, timeline, and go/no-go criteria.

Why this works:- Stakeholders see transparent, weighted trade-offs rather than opinions.- Fast prototypes provide concrete empirical evidence to resolve subjective debates.- Structured documentation gives execs a concise decision with traceable data and mitigations.

Example outcome: After two weeks prototypes showed API+RAG met safety thresholds and shipped MVP in 6 weeks with 40% lower ops risk; long-term plan funded for fine-tuning a private model if cost/usage justify it.

Unlock Full Question Bank

Get access to hundreds of Solution Approach & Modeling Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.