Project Deep Dives and Technical Decisions Questions
Detailed personal walkthroughs of real projects the candidate designed, built, or contributed to, with an emphasis on the technical decisions they made or influenced. Candidates should be prepared to describe the problem statement, business and technical requirements, constraints, stakeholder expectations, success criteria, and their specific role and ownership. The explanation should cover system architecture and component choices, technology and service selection and rationale, data models and data flows, deployment and operational approach, and how scalability, reliability, security, cost, and performance concerns were addressed. Candidates should also explain alternatives considered, trade off analysis, debugging and mitigation steps taken, testing and validation approaches, collaboration with stakeholders and team members, measurable outcomes and impact, and lessons learned or improvements they would make in hindsight. Interviewers use these narratives to assess depth of ownership, end to end technical competence, decision making under constraints, trade off reasoning, and the ability to communicate complex technical narratives clearly and concisely.
HardSystem Design
47 practiced
You are designing a system to serve models with hard real-time guarantees (e.g., p99 < 50 ms). Explain scheduling strategies, model optimizations (compilation, quantization), hardware selection, admission control, and graceful degradation techniques you would implement to meet this SLA reliably.
Sample Answer
Requirements & constraints:- p99 < 50 ms, predictable jitter, mixed request sizes, availability and cost constraints.- Safety: no silent SLA breaches; measurable SLOs and automated enforcement.High-level approach:1) Scheduling & OS-level choices- Use real-time-aware nodes (PREEMPT_RT Linux or RTOS) or isolated CPU cores with IRQ affinity.- Container scheduling: dedicate cores to inference workers; use cgroups and CPU pinning to avoid noisy neighbors.- Request prioritization: prioritized queues (latency-sensitive vs best-effort). Preemptible background tasks run at lower niceness.- Use work-stealing/actor pools sized to match worst-case model latency to avoid queueing spikes.2) Model optimizations- Compile models with TensorRT/ONNX Runtime/OpenVINO for kernel fusion, reduced host-device copies.- Quantization (INT8 / mixed precision) and structured pruning to reduce compute and memory; calibrate to preserve accuracy.- Use operator fusion, batch-norm folding, and AOT compilation for constant folding.- Provide early-exit/cascaded models (small fast model first, heavy model on miss).3) Hardware selection- Choose accelerators with predictable tail latency: inference-optimized GPUs or NPUs with dedicated inference queues (e.g., NVIDIA T4/A100 with MIG, Google TPUs, or edge NPUs).- Favor devices with deterministic scheduling (MIG slices, dedicated inference engines) and sufficient memory to avoid swapping.- High-throughput NICs and NVMe for model loading; colocate model in GPU memory to avoid host transfer.4) Admission control & traffic shaping- Token-bucket or leaky-bucket to smooth bursts; backpressure clients with 429/503 and retry hints.- Dynamic load shedding: when queues exceed thresholds, route to approximate models or degrade quality.- Request routing: hot-model replicas for critical requests; autoscale conservative headroom (reserve capacity) instead of reactive autoscaling.5) Graceful degradation techniques- Model cascade: attempt cheap model -> medium -> full if needed.- Early exit: progressive transformers or classifiers that can return partial results within budget.- Timeout-based fallback: if latency budget passes, return cached or heuristic responses.- QoS knobs: reduce batch size, lower precision, or drop optional features under overload.Operational practices- SLO-driven testing (stress tests targeting p99), continuous profiling of tails, canary deployments, observability: latency percentiles, queue lengths, GPU utilization, memory pressure.- Safety policies: automated scaling + admission control to keep p99 within SLA; alerts when headroom < X%.Trade-offs- Determinism vs cost: reserved capacity increases cost but guarantees SLA; sharing improves efficiency but risks tails.- Accuracy vs latency: quantization/pruning may slightly reduce accuracy; use progressive rollouts and A/B tests.This combination of deterministic scheduling, model-level optimizations, inference-optimized hardware, conservative admission control, and graceful degradation creates predictable p99 behavior while balancing cost and accuracy.
MediumTechnical
45 practiced
Walk through your design for retries, exponential backoff, and circuit breakers on microservices that call model-serving endpoints. Include SLA considerations, timeout choices, priority handling, and how you designed graceful degradation to maintain user experience during partial outages.
Sample Answer
Approach: I design retries, exponential backoff, and circuit breakers to protect model-serving endpoints while preserving SLAs and delivering graceful degradation. The goal: avoid cascading failures, meet latency/availability SLAs, and prioritize user-facing requests.Design summary- SLA targets: p99 latency ≤ 300ms for interactive calls, 99.9% availability. Define an error budget (e.g., 0.1%).- Timeouts: set per-model timeouts tuned to model cost. Example: lightweight classifier timeout = 200–300ms; heavy multimodal model = 2s. Client-side timeout = min(remaining user latency budget, model timeout + small buffer).- Retries & backoff: keep retries minimal to avoid load amplification. Typical policy: max 1 retry for idempotent inference, 0 for non-idempotent. Exponential backoff with jitter: - initial delay = 100ms, multiplier = 2, max delay = 1s, jitter = ±30%. So retry delays = ~100–200ms then ~200–400ms.- Circuit breaker: per-endpoint breaker with sliding window. Example thresholds: trip if ≥5 failures in 60s or error rate >50% in window. Open state duration = 30s; then half-open allowing 1–2 probe requests. Success -> close; failure -> re-open with exponential backoff on open duration.- Priority handling: classify requests as high-priority (interactive) or low-priority (batch/async). Enforce separate queues and rate limits. When backend degrades, reserve capacity for high-priority traffic and shed or queue low-priority.- Graceful degradation / fallbacks: - Tiered model strategy: prefer latest large model; on failure or high latency fall back to smaller, cached, or distilled model with lower cost. - Cache recent responses (per user or per request fingerprint) and return stale-but-valid results within TTL if model unavailable. - Feature-level degradation: return partial results (e.g., metadata, summary, or "best-effort") with clear UX messaging and telemetry hooks.- Observability & automation: - Instrument latency, error rates, retry counts, circuit states, and queue lengths. SLO burn-rate alerts. - Automated scaling: if CPU/GPU utilization high and error budgets safe, scale horizontally; otherwise, trigger degradation policies.- Reasoning: minimal retries prevent amplification on overloaded model servers; jitter reduces synchronized retries; circuit breakers stop repeated attempts during persistent failures; per-priority queues and fallbacks preserve user experience for critical flows.- Edge cases & safety: avoid retrying non-idempotent requests (e.g., stateful logging), ensure fallback model outputs are labeled, and maintain audit logs for degraded responses.This design balances availability and latency while protecting expensive model backends and preserving user-facing experience during partial outages.
EasyTechnical
46 practiced
Explain an access control and privacy measure you implemented for model inference APIs that handled sensitive user inputs. Describe authentication, authorization, encryption-in-transit and at-rest, auditability, and any data-masking or minimization techniques you applied.
Sample Answer
Situation: At my previous company we deployed a text-inference API that could receive user-submitted documents containing PII (names, SSNs, health info). Regulators required strict controls and our customers demanded strong privacy guarantees.Task: I was responsible for designing and implementing access control, encryption, and data-minimization for the inference service.Action:- Authentication: Implemented OAuth2 with short-lived JWTs issued by our identity provider (OIDC). For machine-to-machine calls we used mutual TLS (mTLS) with client certificates rotated via our CA.- Authorization: Enforced RBAC at the API gateway + fine-grained service-level checks (ABAC) inside the model hosting layer so each request was scoped to customer and role (read-only vs. admin). Principle of least privilege applied to service accounts.- Encryption: Enforced TLS 1.3 for all in-transit traffic; internal service-to-service calls also used mTLS. At-rest, model inputs, request metadata, and logs stored in encrypted S3 buckets (AES-256) with keys managed by AWS KMS and access restricted via IAM policies.- Auditability: All inference requests, auth decisions, and model responses (after masking) were logged in an append-only audit log (CloudWatch + immutable storage). Logs captured timestamp, client id, request id, auth verdict, and operator actions. Alerts were configured for anomalous access patterns.- Data masking/minimization: Implemented a preprocessing pipeline that redacted or tokenized PII before sending to the model (regex + ML-based PII detector). For analytics we stored only hashed identifiers and removed raw text after inference unless explicit consent was given. For high-risk outputs we applied response filtering and removed sensitive spans; where needed we applied local differential privacy for aggregate analytics.Result: The service passed security reviews and SOC2 readiness checks. Unauthorized access attempts were blocked and flagged; customer privacy complaints dropped to zero. The approach balanced utility for inference with strong controls to protect sensitive inputs.This design focuses on defense-in-depth: strong auth, least privilege, encryption, auditable trails, and proactive PII minimization.
HardSystem Design
47 practiced
Design a system for nearline personalized recommendations where computing features is expensive. Discuss hybrid architectures combining offline precomputation and online lookups, cache-warming strategies, freshness windows, and how you decide which features to precompute vs compute on demand.
Sample Answer
Requirements & constraints:- Latency target: ~50–200ms per request (nearline).- Throughput: ~10k–100k QPS.- Freshness: some features can be minutes-old, others must be near-real-time.- Feature compute cost: expensive (GPU/large model), some cheap.High-level hybrid architecture:- Offline precompute pipeline (batch): expensive, high-quality features computed periodically (hourly/daily) and materialized to a Feature Store (columnar/key-value store optimized for reads).- Streaming incremental pipeline: event-driven updates for medium-cost features (minutes). Uses incremental updates and delta computation.- Online on-demand compute: low-latency, cheap features or fallback to shallow models computed at request time (CPU/GPU microservices).- Online Feature Cache + Cache Warmers: Redis/Memcached per user or embedding cache in sharded stores; warmed proactively from batch outputs and predictive warmers for hot users/items.- Model serving: ensemble combines precomputed features, streamed deltas, and on-demand features; final scoring service merges inputs and runs lightweight ranking.Deciding precompute vs on-demand:- Precompute if: high compute cost, low update frequency, high reuse (high fanout), deterministic, and tolerant to staleness (e.g., user embedding from nightly training).- On-demand if: highly time-sensitive, personalized to current session context, low compute cost, or combinatorial with request-specific signals.- Hybrid: precompute coarse representation (slow embedding) and refine on-demand with cheap adjustments (context vector, attention over recent events).Freshness windows & policies:- Classify features by freshness SLOs (e.g., near-real-time <1m, soft real-time 1–15m, batch >1h).- Use TTLs per feature and materialized view versioning. Serve most-recent available with metadata indicating staleness.Cache-warming strategies:- Bulk warm after batch jobs: push top-N hot users/items into cache.- Predictive warmers: use access patterns to pre-warm likely-to-be-requested keys (LSTM or heuristic recency/frequency).- Lazy warm + async backfill: on cache miss, return fallback score and asynchronously compute+populate cache, optionally trigger background user notification.- Gradual rollouts to avoid thundering herd: stagger writes and use backoff.Operational considerations:- Monitoring: feature freshness, cache hit rate, end-to-end latency, model quality drift.- Consistency: include version tags and fallbacks to older feature versions during recompute.- Cost/Scaling: separate GPU-heavy offline jobs from CPU online services; autoscale online compute and use quota for expensive on-demand calls.- Trade-offs: more precompute lowers online latency but increases storage and staleness; more on-demand increases compute cost and tail latency.Example flow:1) Nightly job computes user embeddings -> stored in Feature Store and warmed to cache.2) Streaming job updates user activity counters -> merged into store every 5m.3) Request arrives -> score service fetches cached embeddings + latest counters; if session-specific context needed, runs a cheap on-demand transformer to adjust embedding; final rank computed within latency budget.This design balances latency, cost, and freshness by classifying features, combining batch/stream/on-demand compute, and using smart cache-warming and monitoring to maintain quality.
MediumTechnical
58 practiced
You had to choose between a managed model-serving service (e.g., cloud vendor offering) and a self-hosted stack (Kubernetes + TorchServe/TRITON). Walk through the factors you considered (operational overhead, control, scaling, cost, latency, vendor lock-in) and which option you selected and why.
Sample Answer
Situation: I was responsible for deploying a family of transformer and CNN models for a product that needed sub-100ms p99 inference for some endpoints, batch processing for others, and strict data residency for regulated customers.Task: Choose between a managed model-serving offering from our cloud vendor vs. self-hosting on Kubernetes with Triton/TorchServe.Action:- Operational overhead: I estimated SRE time for cluster, GPU drivers, autoscaling tuning, CI/CD, and patching. Self-hosting added ~2 FTEs worth of recurring ops work; managed reduced ops to ~0.5 FTE for integration/monitoring.- Control & customizability: Self-hosting wins — full control over runtime, custom schedulers, custom batching and quantization ops. Managed services had limited custom GPU drivers and no fine-grained scheduling.- Scaling & reliability: Managed service provided easier horizontal scaling and SLA-backed availability. Kubernetes gave more flexible vertical scaling (node pools with different GPU types) but needed careful autoscaler tuning.- Cost: Short-term managed was ~30% higher per inference, but lower personnel cost. At projected scale (10M monthly requests) I modeled a crossover point where self-hosted becomes cheaper after ~9 months when amortizing engineering effort and reserved GPU utilization.- Latency: For the strict low-latency endpoints, colocating models on bare-metal GPU nodes with Triton and customized batching reduced p99 by ~20ms compared to managed service cold-starts. For batch jobs latency wasn’t critical.- Vendor lock-in & compliance: Managed service tied us to API/format and cloud provider features; for regulated customers self-hosting supported on-prem options and easier auditability.Decision & Result:I chose a hybrid approach: use the managed service for non-latency-critical, low-throughput customers to reduce ops burden and speed time-to-market; deploy high-throughput, low-latency models and regulated-customer instances on our Kubernetes + Triton stack. This delivered the p99 SLOs, cut projected long-term cost for heavy workloads, and kept rapid onboarding via managed offerings. Learnings: quantify ops vs. infra cost, run pilot benchmarks for p99 cold-starts, and design CI/CD that can target both serving platforms to avoid lock-in.
Unlock Full Question Bank
Get access to hundreds of Project Deep Dives and Technical Decisions interview questions and detailed answers.
