Deep Technical Expertise and Project Mastery Questions
In-depth exploration of the candidate's most complex or technically challenging project, system, or solution. Interviewers probe the architecture and design decisions involved, the trade-offs weighed among competing approaches, performance and reliability considerations, and the reasoning behind key technology or approach selections. Candidates should be ready to walk through a single complex project from their own experience in detail: describe the problem and constraints, explain the architecture or approach chosen, discuss alternatives considered and why they were set aside, describe the hardest technical challenges encountered, and justify the outcome. Expect pointed follow up questions that test depth of understanding and the candidate's ability to defend their decisions under scrutiny, regardless of the specific technical domain (software systems, machine learning, data infrastructure, customer-facing technical solutions, or another domain the candidate works in).
HardSystem Design
84 practiced
Design a retrieval-augmented generation (RAG) system that integrates a dense vector retrieval service and a large language model across regions. Discuss how to store and partition vector indices, freshness guarantees for newly added/updated documents, replication strategies, routing to the correct index shard/region, and cost vs latency trade-offs when performing synchronous retrieval during inference.
Sample Answer
Requirements & constraints:- Multi-region serving with low-latency RAG (LLM + dense retriever).- Freshness for newly added/updated docs (configurable).- High availability and cost control.- Support large index (billions of vectors).High-level architecture:- Ingest service (APIs for add/update/delete) → Metadata store (global, strongly consistent) + Write-Ahead Log → Regionally-deployed Vector Indexes (sharded) + Background indexer workers → Query path: LLM frontend → Router → Local retriever shard(s) → LLM fusion/generation.1) Index storage & partitioning- Hybrid sharding: - Primary partition by document logical shard (e.g., consistent-hash on doc_id or tenant_id) to localize updates and metadata. - Per-shard vector index uses an ANN algorithm (HNSW for low-latency recall; IVF+PQ for memory efficiency at huge scale).- Each logical shard has one leader replica for writes and multiple read replicas in same region. Shard metadata (range/partition map) stored in global metadata DB (e.g., Spanner/CockroachDB).2) Freshness guarantees- Two modes: - Strong (synchronous): write-through — ingest writes to metadata + leader shard, leader blocks until vector indexed and replicated to N replicas in region. Guarantees immediate visibility for subsequent queries but increases write latency and cost. - Eventual (default): ingest writes to metadata + WAL; leader acknowledges immediately, indexer workers pick up changes and update shard(s) asynchronously. To bound staleness, provide sequence numbers/timestamps and an API for clients to wait-for-indexed(seq_no) or force sync.- For updates/deletes, store tombstones in WAL and apply to indices; versioning to avoid out-of-order application.3) Replication strategy & consistency- Within region: leader-follower synchronous replication for durability of writes (configurable quorum). Read replicas serve queries.- Cross-region: asynchronous replication of vectors and index snapshots. Prefer geo-proximity: keep primary copies in region of origin, replicate compact index deltas to other regions for read-locality.- Use per-shard vector version (monotonic seq) so readers can detect staleness and optionally query origin region for latest vectors.4) Routing to correct index shard/region- Router uses request context (user locale, tenant, or explicit document shard hint) + partition map from metadata store to route to local shard(s).- If local region lacks shard replica or is stale beyond SLA, router can: - Forward to remote region with freshest data (higher latency). - Merge results from local approximate index + one remote precise lookup for freshness-critical queries.- Use consistent hashing + shard map cache at edge; invalidation pushed on rebalances.5) Cost vs latency trade-offs (synchronous retrieval during inference)- Synchronous retrieval for each LLM call adds latency (retrieval + fusion). Options: - Low-latency, higher-cost: maintain warm, in-region read replicas + local ANN indexes (memory-heavy), use HNSW with optimized parameters for recall/latency; pay for more replicas and memory. - Lower-cost, higher-latency: fewer replicas, rely on cross-region async replicas or on-demand cold starts; may increase p99 latency for some requests.- Techniques to balance: - Cached retrieval responses for common queries (LRU/TTL) — reduces calls to index. - Two-stage retrieval: cheap dense ANN to get candidates locally, then optional remote re-rank for freshness-critical items. - Batch retrieval across concurrent requests to amortize cost. - Adjust ANN parameters (ef/search, beam) to trade recall vs latency.- SLA design: e.g., allow 50ms extra for retrieval; if local index can't meet freshness within that, fall back to predictive caching or ask LLM to proceed with warning.Operational considerations- Metrics: recall@k, latency p50/p95/p99 for retrieval and total generation, staleness distribution (secs to index), cost per query.- Rebalance: use re-sharding with background reindex and canary rollouts; ensure consistent hashing minimizes movement.- Backup/restore: snapshot indices and WAL; support rebuild in new regions.- Security: encrypt vectors at rest, secure metadata, per-tenant isolation.Example flows- New doc ingest (eventual): write metadata → WAL ack → background indexer computes vector → updates leader shard → propagate deltas → read replicas refresh.- Query (low-latency): router consults partition map → local ANN search → top-k documents returned with versions → LLM performs fusion. If versions below staleness threshold, router can concurrently request remote fresh candidates and merge.Trade-offs summary- Strong freshness = higher write & storage costs, higher write latency, simpler read correctness.- Eventual = lower cost & write latency, complexity in routing and potential stale answers; mitigated by versioning, wait-for-index APIs, and hybrid local+remote read strategies.This design lets teams configure per-tenant freshness/latency/cost trade-offs, operate large-scale ANN indices across regions, and maintain predictable retrieval SLAs for RAG inference.
MediumTechnical
84 practiced
Design admission control and rate-limiting for a public ML prediction API that must protect expensive models. Policies must support per-user, per-api-key, and per-IP limits, burst allowances, priority tiers, and integration with billing. Explain algorithmic choices (token-bucket, leaky-bucket), distributed enforcement, and trade-offs between accuracy and performance.
Sample Answer
Requirements & constraints:- Multi-tenant controls: per-user, per-api-key, per-IP- Support bursts, steady-rate limits, priority tiers (e.g., paid, free, enterprise)- Integrate with billing (quota metering, overage, throttling to protect expensive models)- Low latency (prediction path), high availability, and accurate enforcement in distributed environmentDesign overview:- Use token-bucket for burst + steady-rate (allows bursts up to bucket size, refills at rate R). Use leaky-bucket (fixed dequeue) when you want smoothing and strict pacing.- Primary enforcement at an edge admission-control layer (API gateway / Envoy filter / custom sidecar) to reject or queue requests before hitting model-serving infra.- Central policy store (Config DB + billing service) holds limits per api-key/user/IP and priority tier. Policies propagated to edge via watch/stream (etcd/Consul/Push config).Algorithmic choices:- Token-bucket per key-tier: tokens refilled at rate R; bucket size = burst allowance. Higher-tier users get larger R and bucket.- Implement weighted tokens for model-cost: different models consume different token counts (expensive model consumes more).- Leaky-bucket used for background queue draining if we want constant output rate to model for capacity smoothing.Distributed enforcement:- Two-level enforcement: 1) Local fast-path at each edge instance maintaining in-memory token-buckets for hot keys (O(1) check) for low latency. 2) Global consistency via a central quota service for strong limits and billing counters. Edges periodically sync counters and fall back to central service when uncertain.- Use probabilistic or approximate counters (Redis with INCR + TTL or Redis Streams) for performance. For stricter guarantees, use Redis with Lua scripts for atomic token consume across cluster, at cost of network hop.- To avoid hot-keys overload, shard keys across Redis clusters and use local cached tokens with refill reconciliation to central store.Billing integration:- Meter token consumption events to billing service asynchronously (Kafka). Map token usage -> billable units (e.g., token weight × model-cost). Enforce hard caps by central quota (reject once central counter shows limit exceeded) or soft caps with grace overage.Trade-offs:- Accuracy vs latency: Local in-memory buckets = very low latency but eventual consistency; central atomic checks = accurate but adds RTT and single-point sensitivity.- Complexity vs cost: Weighted tokens and per-model costing increase fairness but add complexity to policy and metering.- Burst allowance protects UX but risks sudden load; set global circuit-breakers to protect model-serving clusters.- Security: API-key spoofing — authenticate and bind keys to IP/user to prevent circumvention.Edge behaviors:- Priority tiers: when overloaded, use priority queueing—serve high-tier first, ration low-tier via dynamic rate reduction.- Graceful degradation: return 429 with Retry-After, or enqueue in short-lived admission queue with backpressure.Metrics & monitoring:- Track token consumption, rejected requests, latency, per-key errors; alert on hot-key overload and billing anomalies.This design balances low-latency enforcement with global correctness by combining local token-buckets for speed and a central quota/billing service for accurate metering and hard limits.
MediumSystem Design
79 practiced
Design an observability and detection system for model concept drift and data drift in production. Specify which metrics and statistical tests you'd use (e.g., PSI, KS-test), thresholds for alerts, how to combine signal with business metrics, and how automatic retraining triggers or human-in-the-loop processes should be organized.
Sample Answer
Requirements:- Detect data drift (features input distribution) and concept drift (model performance change) in production with low false alarms and clear escalation paths.- Support real-time and batch detection, correlation with business KPIs, automated retrain + human-in-loop.High-level architecture:- Ingest: streaming (Kafka) + batch snapshots into monitoring DB (time-windowed).- Drift Engine: computes statistics, runs tests, persists signals.- Alerting & Dashboard: Grafana/Looker with thresholds, root-cause tooling.- Orchestrator: triggers retrain pipelines (Airflow/Lambda) or human review workflows (Jira/Slack).- Model Registry: stores model versions, metrics, and artifacts.Key metrics & tests:- Input feature drift: Population Stability Index (PSI) per feature (daily/week). Thresholds: PSI < 0.1 OK, 0.1–0.25 watch, >0.25 alert.- Continuous features: Kolmogorov–Smirnov (KS) test with p-value threshold 0.01 for strong drift; complement with Earth Mover’s Distance (EMD) to quantify magnitude.- Categorical features: Chi-square or Jensen-Shannon divergence; frequency shifts > X% flagged.- Latent/embeddings: Multivariate tests (MMD) or classifier two-sample test (AUC > 0.7 indicates drift).- Concept drift (label-based): Rolling window model metrics — accuracy/AUC/precision/recall and calibration (Brier score), and Population Weighted Absolute Error. Use Page-Hinkley or ADWIN for change detection on metric time series.- Prediction distribution shifts: tracking target distribution vs. expected.Alerting thresholds & logic:- Use multi-signal rules to reduce noise: e.g., raise alert when (a) PSI >0.25 on >2 features OR (b) one feature PSI >0.25 AND model AUC drop >0.02 vs baseline OR (c) sustained KS p<0.01 for 3 consecutive windows.- Severity levels: warning (watch), critical (block auto-deploy).Combining with business metrics:- Correlate model metric windows with business KPIs (conversion rate, revenue per user). Run causal checks: if model AUC drop aligns with KPI degradation (statistically significant via change-in-means test), escalate priority.- Compute impact score = expected KPI loss if model underperforms (use uplift model or historical mapping) to prioritize actions.Retraining & human-in-loop:- Two-tier action: 1. Auto-retrain candidate: If data drift detected but business impact low and model candidates validated on holdout, trigger automated retrain pipeline that runs validation, fairness checks, integration tests. If pass, stage new model to canary for limited traffic. 2. Human-in-loop: If concept drift coincides with business-impactful KPI drop or automated validation fails, create investigation ticket with model diagnostics, top contributing features (SHAP), sample cohorts, and require sign-off from ML engineer/product owner before redeploy.- Maintain cooldowns and guardrails: require minimum data size for retrain, monitor shadow-testing for X days, and automatic rollback on negative KPIs.Operational details & trade-offs:- Windowing: use multiple windows (hourly for latency, 7/30-day for trend).- Storage: keep raw samples for sampled replay; privacy & retention policies applied.- Reduce false positives by ensemble of tests and requiring persistence across windows.- Cost vs sensitivity: more strict thresholds reduce risk but increase maintenance; tune thresholds using historical simulation/backtests.This design provides layered detection, risk-based prioritization tied to business impact, and safe automation with human oversight.
MediumSystem Design
79 practiced
Your API must provide on-demand explanations for model predictions, but explanation computation is expensive. Design an architecture to serve explanations with acceptable latency and cost. Consider caching, approximate/heuristic explanations, offline precomputation, gated on-demand explanations, and privacy constraints.
Sample Answer
Requirements & constraints:- Functional: return model prediction + optional explanation API; explanations may be full (accurate, expensive) or fast (approximate/heuristic).- Non‑functional: target tail latency for explanations (e.g., ≤500ms fast, ≤5s full), cost budget, privacy/PII constraints, correctness SLAs.- Usage pattern: sparse requests for full explanations, frequent reads for same inputs.High-level architecture:Client → API Gateway → RouterRouter decides path:1) Fast path: cached / approximate explanation service2) Slow path: gated full-explain workers (on-demand) + async precompute pipelineAlso: Offline batch precomputation, Central cache (Redis + LRU + TTL), Audit & privacy module, Monitoring & quota service.Core components:- Router/Policy Engine: uses request metadata, user level, cost/quota, privacy labels to route: if cache hit → return; else if user allowed & budget → schedule full explain; else compute approximate.- Cache Layer: Redis for hot items (keyed by input hash, model version, explainer version) + object store for large results. Store metadata: computed_at, privacy_flags, accuracy proxy.- Approximate Explainer: fast heuristics (feature importance proxies, SHAP approximations, sample-based LIME with small sample) optimized for latency.- Full Explainer Workers: scalable job workers (K8s) running expensive explainers (Monte Carlo SHAP, integrated gradients) with GPU/CPU pools; expose sync (if within latency) and async endpoints (webhook/poll).- Offline Precomputation: batch jobs over recent training/production inputs (high frequency or high-value segments) to warm cache during low-cost windows.- Gating & Quotas: per-user/per-tenant quotas, cheaper approximate fallback when budget exhausted.- Privacy & Access Control: PII detector that strips sensitive features before storing/serving explanations; differential privacy / noise addition for shared explanations; logging only aggregated metadata.- Monitoring & Metrics: latency, cost per explanation, cache hit rate, accuracy drift.Data flow:1. Request arrives; Router checks cache → if hit and privacy-permitted, return.2. If miss, Router consults policy (user tier, cost). If allowed and time budget ok, dispatch to full worker; else return approximate immediately and optionally enqueue full job for later.3. Full job writes to cache and notifies client if async.Scalability & cost optimization:- Autoscale full workers by queued jobs; use spot instances for batch precompute.- Tiered storage: Redis for hot, S3 for cold.- Use sampling & incremental explanations (coarse→fine); allow progressive rendering to UX.- Cache keys include model & explainer versions to avoid stale explanations.Trade-offs:- Approximate explanations reduce cost/latency but less faithful; must surface confidence/uncertainty to user.- Caching reduces cost but risks stale/PII leakage—mitigated with versioning, TTL, and PII checks.- Precompute increases storage but shifts compute to cheaper windows and improves perceived latency for common inputs.Privacy considerations:- Never cache explanations containing raw PII; store only summaries or differentially private outputs.- Enforce RBAC on explanation retrieval; audit logs with retention limits.- Offer opt-out flags for users whose data shouldn’t be used for offline precompute.Example policy:- Free users: always receive approximate explanation; can request one full explanation per week (async).- Paid users: get N full synchronous explanations/month; others queued async.- High-risk inputs (PII flagged): only approximate + human review.This design balances latency, cost, and fidelity by combining caching, approximations, offline warming, and gated on-demand full explanations while enforcing privacy and monitoring trade-offs.
EasySystem Design
86 practiced
Design a public REST API for a prediction service that returns a predicted probability and top-k explanations. Specify endpoints, request/response JSON schemas (including model version and metadata), authentication approach, error codes, rate-limiting behavior, and how to evolve the API without breaking clients.
Sample Answer
Requirements (brief): single-call REST to get predicted probability + top-k explanations; include model metadata and version; secure and scalable; evolvable without breaking clients.Endpoints- POST /v1/predict - Description: synchronous prediction + explanations- GET /v1/models/{model_id} - Description: model metadata (version, training_date, schema, explanation_method)- GET /v1/health - Description: service health & current model infoRequest/Response JSON schemasRequest POST /v1/predict{ "input": { "type": "object" }, // model-specific feature object (validated server-side) "top_k": 3, // optional, default 3 "request_id": "string", // optional client idempotency/debug "metadata": { "user_id": "string" } // optional, opaque}Response 200{ "request_id": "string", "model_id": "string", "model_version": "2025-11-01-rc1", "timestamp": "2025-11-22T14:12:00Z", "prediction": { "label": "positive", "probability": 0.87 }, "explanations": [ { "feature": "age", "contribution": 0.12, // positive raises probability by 0.12 "score": 0.45, // raw feature importance "method": "shap", "confidence": 0.95 } ], "model_metadata": { "description": "Fraud risk v2", "explanation_method": "shap", "training_data_hash": "sha256:..." }}Errors and codes- 400 Bad Request — validation errors (include field errors array)- 401 Unauthorized — missing/invalid token- 403 Forbidden — insufficient scope/quota- 404 Not Found — model or endpoint not found- 422 Unprocessable Entity — input schema OK but semantics invalid- 429 Too Many Requests — rate limit exceeded- 500 Internal Server Error — server-side failure (include trace_id)Authentication- Primary: Bearer JWT with scopes (predict, metadata). Validate signature and claims (iss, exp, sub, scopes).- Alternative for short-term: API Key in Authorization: ApiKey <key>- TLS required (HTTPS). Use OAuth2 client credentials for server-to-server flows.Rate limiting & headers- Per-API-key/user token-bucket or leaky-bucket (e.g., 100 req/min default).- Return headers: - X-RateLimit-Limit: 100 - X-RateLimit-Remaining: 73 - X-RateLimit-Reset: 167 (seconds)- Offer higher tiers via negotiated SLA.Observability & retry guidance- Include trace_id in responses/error for support.- For 5xx, clients may retry with exponential backoff and jitter; for 429 use Retry-After header.Evolving the API without breaking clients- URI versioning (v1 in path). Follow semantic versioning for major changes (v2) that break contract.- Backward-compatible changes: add optional response fields, extend model_metadata; ensure defaults if absent.- Capability discovery: GET /v1/models/{id} returns supported explanation methods, response schema version.- Use response content negotiation (Accept: application/vnd.myapi.v1+json) for advanced clients.- Deprecation policy: provide Deprecation and Sunset headers with timeline, and warn in model_metadata/predict response for deprecated features.- OpenAPI spec published; maintain changelog. Run consumer-driven contract tests and provide transition docs and migration examples.Implementation notes- Validate input schema server-side; return structured field errors.- Sign and timestamp explanations (or include explanation_version) so clients can audit/regenerate if model changes.- Consider async /batch endpoint for long-running or heavy explainers: POST /v1/predict-async.
Unlock Full Question Bank
Get access to hundreds of Deep Technical Expertise and Project Mastery interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.