Technical Project Stories Questions
Prepare two to four hands on technical project narratives that demonstrate engineering depth, architectural thinking, and measurable outcomes. For each project describe the business problem, system architecture or design choices, trade offs evaluated, scaling and reliability challenges, instrumentation or observability decisions, implementation details and technologies used, your specific responsibilities, and the measurable results achieved. Be prepared to dive deep on technical decisions, show diagrams or component flows if asked, describe how technical debt and operational run book items were managed, and explain how the work influenced broader engineering practices. Include examples across front end, back end, infrastructure, data, and security as relevant to the role.
HardSystem Design
40 practiced
Describe a project where you designed a secure end‑to‑end deployment pipeline for models handling PII. Include secrets management, least‑privilege access controls, key management for encryption at rest and in transit, audit logging and tamper evidence, data access review processes, automated drift detection for sensitive attributes, and compliance evidence you produced.
Sample Answer
Requirements & constraints:- Models train/serve on datasets with PII; must ensure confidentiality, integrity, least privilege, provable audit trail, automated detection of sensitive-attribute drift, and produce compliance artifacts (e.g., for SOC2/GDPR).High-level architecture:- Data lake (encrypted S3/GCS) → training infra (Kubernetes/GPU cluster in VPC) → CI/CD (GitHub Actions / Jenkins) → model registry (MLflow / Artifact repo) → serving (K8s/managed inference) - Centralized secrets & keys: HashiCorp Vault + cloud KMS/HSM for master keys- Logging & audit: Immutable audit log (CloudTrail / Auditd → WORM storage + signing)Core components & responsibilities:1. Secrets & Key Management- Store API keys, DB credentials, service principals in Vault with dynamic secrets for DB connections (short TTL).- Use cloud KMS backed by HSM for envelope encryption: data keys per dataset encrypted with KMS; rotating CMKs quarterly; retain key rotation policy and logs.- Use asymmetric keys for signing model artifacts to ensure provenance.2. Least-Privilege Access Controls- IAM roles per service with scoped permissions (read-only training data, write-only model registry, no direct key access).- Kubernetes: pod-level service accounts, network policies restricting egress/ingress, secrets projected via CSI driver.- Approval workflows: any role granting data access requires manager + data owner approval recorded in IAM ticketing system.3. Encryption in transit & at rest- TLS 1.2+ with mTLS between services; internal certs managed via private CA (Vault PKI).- All storage (S3/GCS/EBS) encrypted using envelope encryption; backups also encrypted.4. Audit logging & tamper evidence- Centralized immutable logs: send system logs, Vault audit, KMS key-use logs to append-only WORM bucket with SHA-256 manifest and periodic Merkle-tree signatures stored in Vault; alerts on unexpected log deletions.- Sign every release artifact (model binary + metadata) with private key; signatures validated at deploy time.5. Data access review processes & compliance evidence- Monthly automated access review report: list of principals with dataset access, generated from IAM and Vault, digitally signed, stored in compliance repo.- Produce artifacts per release: data manifest (hashes, lineage via DVC), training job config, model card (purpose, intended use), signed provenance, access logs, key-use logs, and DRP/retention attestation.- For GDPR: dataset pseudonymization record and data minimization justification included.6. Automated drift detection for sensitive attributes- During serving, telemetry captures distributions of sensitive attributes (e.g., age bracket, zipcode prevalence) in aggregated privacy-preserving buckets (differential privacy / k-anonymity thresholds).- A monitoring pipeline computes population-level statistical divergence (KL divergence, population stability index) and runs concept-drift detectors on sensitive-feature embeddings.- Alerts when drift exceeds thresholds trigger gated retraining and mandatory data-owner review; retraining locked behind manual approval if sensitive-attribute impact > threshold.Data flow & CI/CD:- PR triggers pipeline: static checks, unit tests, data-slice tests (including PII-presence checks), training in isolated ephemeral infra with least-privilege role; artifacts stored in registry with signed metadata; deployment requires automated policy checks (policy-as-code: OPA/Rego) validating encryption, key usage, model signature, and results of sensitive-attribute tests. Any failing policy blocks promotion.Scalability & reliability:- Use autoscaling GPU pools and asynchronous telemetry ingestion (Kafka). KMS calls cached (never expose plaintext keys). Audit storage lifecycle policies move logs to archived encrypted WORM with retention policies.Trade-offs:- HSM-backed KMS increases cost but provides stronger attestation required for compliance. Dynamic secrets add complexity but reduce long-lived credentials. Differential privacy limits fidelity of telemetry; selected balance to preserve monitoring utility while protecting individuals.Why this design:- Provides defense-in-depth (network, IAM, secrets, keys), end-to-end provenance and tamper evidence (signing + immutable logs), automated sensitive-drift detection with human-in-the-loop controls, and produces concrete compliance artifacts (signed access-review reports, dataset manifests, model cards, key-use logs) ready for audit.
MediumSystem Design
51 practiced
Tell a story about designing a hybrid batch + online inference system to balance cost and latency. Describe which features were precomputed in batch, which were retrieved online, routing logic to pick the path, freshness semantics for each, how you validated equivalence between paths, and how you monitored divergence over time.
Sample Answer
Requirements:- Serve inference requests at 5–50k QPS with 50–200ms P95 latency target.- Minimize cost by precomputing expensive, slowly-changing features; keep recent, user-specific features online.- Maintain correctness: responses from batch path should approximate online path within SLA.- Provide observability and automated routing safety.High-level design:- Two-path hybrid: Batch path (cheap, cached, precomputed features + model inference) and Online path (real-time features, fresh model or lightweight delta-correction).- Router decides per-request path based on user/account activity, feature freshness needs, and cost budget.Which features precomputed in batch:- Heavy aggregated statistics (30/60/90-day rolling metrics), embedding shards from nightly large-scale retraining, global item popularity, historical user-item interaction embeddings, slow-to-change metadata (category, static item signals).- Precompute model-score baselines for common queries (top-K per user cohort) and store in feature store / cache (Parquet -> Redis/Bigtable).Which features retrieved online:- Very recent interactions (last 1–24 hours), session context, ephemeral signals (clicks, dwell time), real-time inventory/availability, A/B flags, ML features that change per request (current device, geolocation).- Lightweight online model to compute deltas or apply short-term correction to batch baseline.Routing logic:- Rule-based + dynamic policy: - If user has recent interactions within window W (e.g., 1 hour) or request requires hard-freshness flag → route to online path. - If user is low-activity and request matches cached cohort → route to batch path. - Budget-aware sampling: stochastic routing (e.g., 90% batch, 10% online) for eligible requests to control cost. - Fail-safe: if batch cache miss, fallback to online.Freshness semantics:- Batch features: “eventual consistent” with TTL = batch cadence (e.g., 6–24 hours). Annotated with timestamp and staleness.- Online features: real-time; staleness < few seconds.- Responses include provenance metadata (path, feature timestamps); model applies staleness-aware weighting (decay factors).Validating equivalence between paths:- Shadowing: duplicate a subset of live requests — run both batch and online pipelines in parallel; do not serve batch results unless safe.- A/B and canary tests: initially compare metrics on a holdout traffic slice; compute per-request score difference, ranking delta (NDCG), and business KPIs.- Offline replay: store request logs and replay through both pipelines to compare outputs deterministically.Monitoring divergence over time:- Metrics: distribution of score differences, top-K overlap, NDCG, calibration drift, feature staleness histograms, percent of requests routed to each path, latency and cost per path.- Alerts: thresholds on mean/95th score delta, drop in overlap > X%, unusual increase in batch cache misses.- Drift detection: run periodic model-sanity checks (KL divergence on feature distributions) and automated retrain triggers if drift exceeds threshold.- Visualization: dashboards showing per-cohort divergence, heatmaps by feature staleness.Example validation numbers:- Acceptance criteria: mean score delta < 0.02, top-10 overlap > 85%, no significant KPI regression in 7-day A/B.- If divergence > threshold, temporarily increase online routing for affected cohort and schedule accelerated batch recompute.Trade-offs and learnings:- Precompute reduces cost and tail latency but introduces staleness risk; hybrid routing + delta-correction balances trade-offs.- Observability and automated safety (shadowing, fallbacks) are essential before shifting production traffic to batch path. Continuous monitoring and small-step rollouts prevent user impact.
MediumSystem Design
47 practiced
Tell a story about designing a streaming API for real‑time AI features (for example, live transcription or real‑time scoring). Discuss protocol choice (gRPC vs WebSocket vs HTTP/2), backpressure and flow control, partial response semantics, latency vs throughput trade‑offs, error handling and reconnection strategies, and how you instrumented end‑to‑end latency.
Sample Answer
Situation: I designed a streaming API to power live transcription and real‑time scoring for a video conferencing product—requirements were sub-200ms per-token latency for captions, sustained multi-minute sessions, and fairness across many concurrent users.Requirements & constraints:- Bi-directional low-latency streaming for audio in and tokens/out- Backpressure so slow clients don’t overload GPU inference- Partial/ interim results with update semantics- Tolerant to intermittent network and able to resume sessions- Instrumentation for end-to-end latency (P50/P95/P99) and observabilityProtocol choice- I chose gRPC over HTTP/2 (bi‑directional streaming) because it gives multiplexed streams, built‑in flow control, small binary protobuf frames, and native support across server languages. WebSockets is viable for browser-first deployments; in that case we used a thin translation gateway that converted WebSocket clients into internal gRPC streams. Plain HTTP/1.1 SSE was insufficient for bi‑directional needs.Backpressure & flow control- Leveraged HTTP/2 windowing and gRPC flow control to avoid memory blowups. At the application layer implemented a token-bucket: the server only accepts N concurrent inference requests per client (configurable per plan). The inference worker returns tokens with each response indicating remaining capacity. When capacity is exhausted, server replies with a “slowdown” control frame (protobuf message) so clients lower audio chunk frequency or reduce sample rate. For browser clients we provided adaptive client-side capture: reduce chunk size, switch to compressed codec, or drop non-speech segments.Partial response semantics- Designed messages with three fields: type {partial, final, control}, sequence_number, and delta. Partial messages contain interim text + confidence + stable_prefix_len to indicate which prefix is final. Clients display partial text optimistically but replace only when sequence or stable_prefix advances. For scoring, partial scores are tagged as provisional; final messages include final score and optional alignment tokens. This avoids flicker and supports idempotent replays.Latency vs throughput trade-offs- Two modes: low-latency (small audio frames, immediate model decode with greedy or low‑beam search) and high-throughput (larger frames, batched GPU inference, higher beam). Exposed mode hints in the session init. Default used low-latency decoding with sequence-level batching (aggregate up to 8 concurrent requests every 10–20ms) to keep GPUs busy without adding user-visible delay. Tuned batch sizes empirically to balance GPU utilization and 99th percentile latency.Error handling & reconnection- Use transient vs permanent error model. For transient errors (RESOURCE_EXHAUSTED, UNAVAILABLE), client implements exponential backoff with jitter and session resume token (last-acked sequence_number). Server persisted minimal session state (last sequence, model checkpoint id, partial hypothesis) in a fast cache (Redis) for short-lived resumes. For permanent errors (INVALID_ARGUMENT, UNAUTHENTICATED) return deterministic code and human-readable message. For network reconnects we supported resumable streams: client resends from last acked chunk id; server dedup by sequence_number. For long sessions we issued periodic heartbeats and automatic session checkpoints to allow resumption after longer interruption.Instrumentation & end-to-end latency- Instrumented with distributed tracing (OpenTelemetry): client stamps request with traceparent and local capture timestamp; server appends model decode start/end timestamps and inference GPU enqueue/dequeue times. We computed client-to-server, server-queue, model-latency, and server-to-client RTT. Metrics: counters for bytes/sec, histograms for per-token latency (P50/P95/P99), GPU batch sizes, queue lengths, and backpressure events. Synthetic load tests and shadow traffic validated tail latencies. Alerts on P99 > SLA and elevated backpressure events triggered autoscaling of inference workers and investigation.Result & learning- Final system delivered consistent sub-200ms interactive captions for 95% of sessions and graceful degradation under contention. Key lessons: align protocol capabilities with client types (gRPC internal, WebSocket gateway for browsers), push flow-control decisions to both transport and app layers, and design clear partial/final semantics so downstream UI can be simple and robust.
HardSystem Design
40 practiced
Deep dive into a case where you designed cross‑region failover for an AI service that maintained stateful user sessions. Explain session replication strategies (active‑active, primary/secondary), sticky routing vs session transfer, conflict detection and resolution, trade‑offs in read/write locality, and how you tested end‑to‑end failover under production‑like load.
Sample Answer
Requirements & constraints:- Stateful AI service with per-user sessions (context, embeddings, partial outputs), global availability across two regions, RPO ~seconds, RTO <30s, low latency for active users, strong consistency for writes where possible.High-level design:- Primary/secondary with optional active-active for read-heavy users. Each region runs full inference stack + session store. A global load balancer routes users to nearest region by latency/geo and health.Session replication strategies:- Primary/Secondary: Session owner lives in one region; writes are synchronously persisted locally and asynchronously replicated to secondary with durable WAL and change-log (Kafka/CDC). Guarantees local write latency; failover promotes secondary using last-applied log index.- Active-Active: Both regions accept reads/writes; use CRDT-like structures for conversational history (append-only logs) and vector-clock metadata. Writes are merged deterministically; conflicts resolved via timestamps + causal ordering.Sticky routing vs session transfer:- Sticky routing (LB session affinity) minimizes cross-region latency and conflicts for long-lived sessions.- Session transfer performed on failover: transfer the session by promoting secondary and updating LB. Transfer stream includes WAL snapshot + in-flight operations; resume inference after replay.Conflict detection & resolution:- For primary/secondary: detect gaps via sequence numbers; apply missing ops in order.- For active-active: detect conflicting edits via vector clocks. Resolution policy: append-only conversation merges; for state mutations (e.g., model persona switch), use last-writer-wins with causal validation or require application-level merge (human-in-loop) for ambiguous state.Read/write locality trade-offs:- Local writes: low latency, weaker global consistency unless sync-replicated (higher latency).- Global consistency: synchronous multi-region commit (Paxos) gives consistency but adds 100+ ms and complexity.- Chosen hybrid: prefer local writes with async replication for throughput; critical ops optionally use synchronous commit.Testing end-to-end under production-like load:- Create realistic session workloads (conversation length, embedding ops, model infer calls) with traffic replay and synthetic users across regions.- Inject failover scenarios: network partition, full region shutdown, leader crash mid-write.- Validate invariants: no lost messages, deterministic merge results, RTO/RPO met.- Use chaos testing (Chaos Monkey), golden-data checksums, and trace-based verification to ensure session continuity and consistent model state after failover.- Monitor metrics: replication lag, sequence gaps, tail latency, error rates; run drills with rollback plans.Outcome & lessons:- Hybrid primary/secondary with append-only logs gave best latency/consistency balance for conversational AI. Active-active used only for read-heavy analytics sessions. Emphasize observable replication state and clear promotion protocols to avoid split-brain.
HardSystem Design
38 practiced
Provide a project‑level account of designing a distributed feature service (not a full data pipeline) to support low‑latency feature retrieval for models. Cover API design (sync lookup), consistency model, caching and invalidation, schema evolution strategies, versioning of features, and how you tested the system for scale and correctness.
Sample Answer
Requirements (clarified): low‑latency (<5ms P50 lookup), high QPS (100k‑1M/s), per‑entity key lookup for online models, support batched sync API, strong read availability, eventual consistency acceptable for non-critical features, schema/versioning and safe evolution, TTL/invalidation controls, monitoring and auditability.High-level architecture:- Frontend: gRPC/HTTP sync lookup service that accepts batch(keys, feature_set, version) and returns feature vectors + metadata.- Router/Load‑balancer: directs to appropriate shard/replica.- Storage: mix of in‑memory distributed cache (Redis Cluster / Aerospike) + persistent scalable KV (Cassandra/Bigtable) as source of truth.- Control plane: schema registry, feature metadata store, version manager, ingestion worker APIs.API Design (sync lookup)- Endpoint: Lookup(batch_keys[], feature_set_name, feature_version, timeout_ms, prefer_fresh=true)- Response: { key, feature_vector, feature_version, timestamp, source: CACHE|SOT, stale_flag, checksum }- Semantics: All or partial responses allowed with explicit error codes; include per‑field nullability and provenance.Consistency model- Read‑heavy: use read‑through cache with configurable staleness policy.- Default: read-after-write eventual consistency (writes propagate async). For critical features, support synchronous write‑through or strong consistency via quorum reads from storage.- Provide a "fresh" flag to force bypassing cache for model evaluation or debugging.Caching & invalidation- Read‑through + write‑through patterns; cache keys include feature_set+version+key.- TTLs per‑feature; active invalidation via pub/sub on control plane when features updated (broadcast delta invalidation). Use versioned keys so updates avoid race conditions.- Background async refresh (lazy warms) and hotkey protection (request coalescing, token bucket).- Soft deletes: mark tombstones with propagation window to avoid stale reappearance.Schema evolution & versioning- Schema registry with backward/forward compatibility checks. Allow additive, nullable changes without forcing recompile.- Feature sets are versioned; each lookup specifies version; default resolves to latest stable.- Migration: dual‑write to old+new versions during rollout, canary reads by percentage, A/B routing for model validation.- Transformation layer supports on‑read computed features with deterministic functions stored in metadata.Testing for scale & correctness- Unit tests for serialization, null handling, provenance, checksum.- Integration tests: end‑to‑end with mock ingesters and storage.- Load testing: synthetic key distributions (uniform, zipfian, hotkeys) with k6/jMeter and chaos scripts. Measure P50/P95/P99, tail latencies, error rates.- Consistency tests: inject writes, simulate partial propagation, verify versions and tombstone propagation with randomized property testing.- Fault injection: node failures, network partitions, cache flushes; verify graceful degradation and failover.- Continuous monitoring: dashboards (latency, hit rate, staleness), alerts, and periodic data drift checks.Key trade-offs- Strong consistency adds write/read latency; use selectively.- Versioned keys simplify invalidation but increase storage; mitigate via TTL and compaction.- Favor predictable low tail latency via caching and request coalescing rather than synchronous SOT reads.
Unlock Full Question Bank
Get access to hundreds of Technical Project Stories interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.