Architecture and Technical Trade Offs Questions
Centers on system and solution design decisions and the trade offs inherent in architecture choices. Candidates should be able to identify alternatives, clarify constraints such as scale cost and team capability, and articulate trade offs like consistency versus availability, latency versus throughput, simplicity versus extensibility, monolith versus microservices, synchronous versus asynchronous patterns, database selection, caching strategies, and operational complexity. This topic covers methods for quantifying or qualitatively evaluating impacts, prototyping and measuring performance, planning incremental migrations, documenting decisions, and proposing mitigation and monitoring plans to manage risk and maintainability.
MediumSystem Design
35 practiced
You have a nearest-neighbor embedding service used by recommendations. Propose a multi-tier caching and indexing strategy (RAM cache, approximate nearest neighbor index, cold-storage re-compute) to reduce GPU usage. Discuss freshness, eviction policies, and how to measure staleness impact on recommendations.
Sample Answer
Requirements & constraints:- Low GPU use for serving nearest-neighbor (NN) lookups at high QPS (<100ms tail).- Freshness: embeddings update as user/item models retrain hourly → near-real-time preferred but eventual acceptable.- Scale: millions of vectors, memory limited.High-level multi-tier design:1) RAM hot-cache (per-serving node, in-memory KV) - Stores: precomputed top-K results or pre-filtered candidate lists for the most-active queries/users. - Tech: Redis (maxmemory-policy volatile-lru) or in-process LRU cache for ultra-low latency. - Use TTLs based on update cadence (e.g., 10–30 minutes) and dynamic hotness scoring.2) ANN index layer (approximate search on CPU) - Stores: global ANN index (HNSW, FAISS IVFPQ) on CPU RAM or pooled servers. - Serves fast approximate NN without GPU; periodically rebuilt/merged. - Sharding by vector partition, with replication for availability.3) Cold-storage re-compute (GPU-backed) - Runs batch re-embedding, full index rebuilds, and heavy exact re-ranking when necessary. - Triggered by model retrain, large data drift, or cache-miss budget thresholds.Data flow:- Request → check RAM hot-cache; hit => return.- Miss → query ANN CPU index for candidates → lightweight re-rank (CPU) → return and populate RAM cache.- If ANN confidence low or strict SLA for quality breached → fall back to GPU re-rank or exact NN.Freshness & eviction:- Multi-dimensional eviction: LRU for recency, LFU for frequency, and explicit invalidation on model/schema changes.- TTLs tiered: hot-cache short TTL (10–30m), ANN index time-windowed snapshots (hourly/daily), cold-storage authoritative.- Use invalidation streams: when embeddings update, publish keys for targeted invalidation; schedule full index rebuilds during low traffic.Measuring staleness impact:- Instrument A/B tests: compare recommendations using fresh GPU-computed embeddings vs cached/ANN results on metrics (CTR, conversion, retention).- Compute embedding-age metric per response and correlate with downstream KPIs.- Offline proxy: simulate lookups using historical embeddings to estimate degradation (NDCG/Recall@K vs ground-truth).- Define acceptable thresholds (e.g., <2% drop in CTR); when exceeded, raise cache purge or force re-compute.Trade-offs:- ANN reduces GPU cost but may slightly reduce quality; mitigate with periodic re-ranking and confidence scoring.- More aggressive TTLs improve freshness but raise GPU/index rebuild frequency.- Cost/latency balance tuned via hotness-driven caching and staged fallbacks.This design reduces GPU pressure by serving most traffic from RAM/CPU ANN while maintaining quality via targeted GPU re-computes and measurable staleness detection.
HardSystem Design
37 practiced
Architect a real-time personalization service that must serve 100k users with p95 latency <150ms, support frequent online feature updates, and provide both global and per-user context. Describe choices for feature storage, cache consistency, model locality, and trade-offs between freshness and latency.
Sample Answer
Requirements & constraints- Serve 100k users with p95 latency <150ms for personalization decisions.- Frequent online feature updates (near-real-time).- Need global context (campaigns, trends) and per-user context (session, profile).- Freshness vs latency trade-offs must be explicit.High-level architecture1. Online feature layer (low-latency): per-serving-node local cache (in-memory) + global distributed cache (Redis Cluster / KeyDB).2. Persistent online store: a highly-available KV DB for authoritative features (DynamoDB / CockroachDB / Cassandra) for writes and rare cold reads.3. Change stream pipeline: CDC/stream (Debezium/Kafka) → stream processors (Flink/ksql) to compute derived features and push updates to caches.4. Feature store coordination: Feast-like control plane for schema, freshness policies, and offline/online sync.5. Model serving: model replicas colocated with local cache (model locality) for sub-150ms inference; fallback to central scorer if needed.6. Monitoring & consistency: metrics, alerting, and a versioning system for features/models.Feature storage choices- Per-user volatile/session features: stored in local in-memory cache (LRU) on serving nodes for microsecond access. Persisted asynchronously to KV DB.- Stable user attributes & derived features: authoritative in distributed KV DB; updated via stream.- Global context: stored in Redis with sharding; small, frequently-read aggregates. Large vectors/embeddings in vector DB (Milvus) if needed.- Offline features: object storage + batch feature store for training.Cache consistency strategies- Write-through for critical updates: write to KV DB and update Redis atomically (via transactions or Redis Streams) to keep strong-ish consistency for important features.- Event-driven invalidation/update: CDC → Kafka topics; consumers push updates to per-node caches (push-based) to keep caches fresh without polling.- Versioned features & TTLs: include feature-version and timestamp; serving logic rejects stale versions beyond staleness threshold.- Read-after-write needs: route to authoritative KV DB or use synchronous cache update on the write path.Model locality & serving- Co-locate model with local cache on the same serving instance to avoid network hops: model reads local cached features and global Redis for small aggregates. This yields the best latency.- Keep model size bounded for in-memory serving (quantize/optimize); use GPU/CPU depending on model cost.- Central scorer fallback for cold-starts or heavy models (accept higher latency, used rarely).Freshness vs latency trade-offs- Immediate strong consistency: write-through + sync updates → freshest results but higher write latency and throughput pressure.- Eventual consistency with push updates and TTLs → much lower inference latency (meets p95) but allows brief staleness window. Mitigate by: - Prioritizing features by freshness requirements (strict vs soft). - Using hybrid updates: critical features use synchronous updates; non-critical use async streams. - Feature versioning and confidence decay: model accounts for feature age.Scalability & capacity planning- Estimate QPS from active users; shard Redis and KV DB; autoscale serving pool.- Push-heavy updates: partition streams by user ID to avoid hotspots.- Use consistent hashing for cache locality.Failure modes & mitigation- Cache misses → authoritative KV DB read (higher latency) and backfill to cache.- Stream lag → detect via watermarking; degrade gracefully with cached defaults.- Model rollback/versioning for bad updates.Summary (operational rules)- Classify features by freshness SLAs and store accordingly.- Use event-driven cache updates for scalability and near-real-time freshness.- Co-locate compact models with local caches to meet p95 <150ms, and accept selective synchronous updates for critical features to balance freshness/latency.
HardTechnical
27 practiced
Evaluate trade-offs between using managed cloud AI inference services (e.g., SageMaker, Vertex AI) vs self-managing inference infrastructure for a fast-growing startup that needs to balance speed-to-market, cost, and control. Include migration, locking risks, and monitoring/observability differences.
Sample Answer
Start by restating priorities: for a fast-growing startup the three axes are speed-to-market (deliver features fast), cost (both short-term and TCO as scale grows), and control (performance tuning, security, custom hardware). Trade-offs depend on which axis is primary now vs later.High-level comparison- Managed inference (SageMaker/Vertex AI): - Speed-to-market: very high. Built-in deployment pipelines, autoscaling, A/B rollout, model registry, and integration with data stores reduce engineering weeks/months. - Cost: higher per-inference and less granular control; but reduces staffing cost and operational overhead. - Control: limited—less ability to optimize custom runtimes, scheduler behavior, or exotic GPUs. - Migration/lock-in: medium — APIs are cloud-specific (endpoint formats, model registry). Exporting models is possible but re-implementing deployment logic can be non-trivial. - Monitoring: strong out-of-the-box telemetry (latency, error rates, resource metrics) and managed alerts, but limited custom instrumentation deep inside runtimes.- Self-managed inference (Kubernetes + KFServing/Triton on GKE/EC2/on-prem): - Speed-to-market: slower initially—needs infra, CI/CD, autoscaling, canary logic built or configured. - Cost: potentially lower at scale due to spot instances, reserved capacity, right-sized hardware; higher ops staffing cost. - Control: maximal—custom runtimes, mixed precision, batching, inference optimizations, custom hardware choices. - Migration/lock-in: lower vendor lock-in if using standardized containers and open tools, but vendor-specific optimizations (e.g., instance types) still matter. - Monitoring: fully customizable (Prometheus + Grafana + eBPF/tracing) enabling model-level attribution, custom feature drift, and advanced observability, but requires build/maintenance effort.Migration and lock-in considerations- Start managed for MVP: ship fast, validate product-market fit, use managed model registry and endpoints.- Design for portability: containerize models, separate model artifact format (ONNX/TensorRT when possible), codify deployment via Terraform/Helm, keep infrastructure-as-code and CI pipelines cloud-agnostic.- Exit plan: maintain lightweight self-managed staging environment to exercise deployment scripts; periodically test export/redeploy path (runbook).- Lock-in mitigation: avoid proprietary SDKs for inference logic; use standardized logging and metrics schema; store model artifacts in neutral formats and version control.Monitoring & observability differences- Managed: quick visibility (SLA metrics); use provider logs + built-in dashboards; supplement with application-level tracing by adding hooks to request pipeline.- Self-managed: full-stack observability—Prometheus metrics, OpenTelemetry traces, model input/output sampling, feature-distribution drift detectors, and offline batch validation. Requires SLOs, error-budget policies, and automated rollback.Recommended phased plan1. MVP (0–6 months): Managed endpoints for rapid delivery; set up CI to produce containerized artifacts; implement basic telemetry and cost tracking.2. Scale validation (6–18 months): Measure cost per inference, latency percentiles, ops pain points. Implement portability tests: export container + ONNX and deploy to a small self-managed cluster.3. Decision gate: if 6–12 month forecast shows costs or control constraints exceeding threshold, build full self-managed infra with reuse of deployment artifacts and observability pipelines. Keep managed for low-critical or bursty workloads.Practical mitigations- Use open standards (ONNX/TensorRT), containerize, codify infra, create runbooks for migration, and instrument models with consistent metrics. Track per-model cost, latency, and error SLAs to decide when to shift workloads.Recommendation- For a fast-growing startup: start with managed to maximize speed, but engineer explicitly for portability and visibility. Transition selectively (latency-critical or high-volume models) to self-managed when costs or control needs justify the ops investment.
EasyTechnical
32 practiced
Technical: Implement a thread-safe token-bucket rate limiter in Python that supports burst capacity, refill rate (tokens/sec), and a non-blocking `allow_request(key)` API suitable for per-user inference throttling. Explain assumptions and how you'd extend this to a distributed environment.
Sample Answer
Approach: use a per-key token-bucket tracked in memory. Each bucket stores (tokens, last_ts). On each allow_request(key) call we atomically (thread-safe) refill based on elapsed time, cap at burst capacity, then consume 1 token if available and return True/False. Use per-key locks (or a single lock for simplicity) to be safe under threads. Non-blocking: returns immediately.Key points:- Uses time.monotonic() to avoid clock jumps.- Per-bucket lock keeps contention low for many users.- Non-blocking: allow_request returns immediately.Complexity:- Time: O(1) per request.- Space: O(U) for U active users.Edge cases:- Large number of distinct keys -> memory growth; use eviction (LRU) for inactive keys.- Very high rate -> floating precision; consider using integers or microsecond units.- Burst capacity 0 -> effectively blocks all.Distributed extension:- Use a central store with atomic operations, e.g., Redis + Lua script implementing token refill & consume in one atomic step. The Lua script computes refill based on stored last_ts and tokens, caps at capacity, decrements if possible and returns allowed boolean. This avoids race conditions across processes. Consider clock skew by using server-side time (TIME command) or monotonic server timestamp. For extreme scale, use local token caches for low-latency checks and periodically reconcile with Redis (sharded by key). Trade-offs: central Redis adds network latency and single point of failure; mitigate with clustering and replication.
python
import time
import threading
from collections import defaultdict
class TokenBucket:
def __init__(self, rate, capacity):
"""
rate: tokens per second
capacity: max tokens (burst capacity)
"""
self.rate = float(rate)
self.capacity = float(capacity)
self.tokens = capacity
self.last_ts = time.monotonic()
self.lock = threading.Lock()
def allow(self, tokens=1):
with self.lock:
now = time.monotonic()
# refill
elapsed = now - self.last_ts
self.last_ts = now
self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
if self.tokens >= tokens:
self.tokens -= tokens
return True
return False
class RateLimiter:
def __init__(self, rate, capacity):
self.rate = rate
self.capacity = capacity
self.buckets = defaultdict(lambda: TokenBucket(rate, capacity))
self.global_lock = threading.Lock() # protects creation race
def allow_request(self, key):
# ensure bucket exists in a thread-safe way
with self.global_lock:
bucket = self.buckets[key]
return bucket.allow(1)MediumSystem Design
35 practiced
Design a scalable model-serving architecture to support two workloads: (1) low-latency online inference with 200ms p95 SLO at 5k QPS, and (2) high-throughput batch scoring of 10M requests/day. Outline components (load balancer, pre/post processors, model servers, cache, autoscaler), trade-offs between separation vs shared infra, and a monitoring strategy.
Sample Answer
Requirements & constraints:- Online: 5k QPS, 200ms p95 SLO, tight tail latency.- Batch: 10M requests/day (~116 rps averaged, but large spikes of throughput acceptable), cost-sensitive.High-level architecture:- Global Load Balancer / API Gateway: routes traffic to online inference endpoints; provides TLS, auth, rate limiting, per-customer routing.- Ingress layer -> Pre-processing service (lightweight, autoscaled per request; validates, normalizes, tokenization).- Feature cache (Redis/Memcached): hot features, embeddings, and recent responses to serve cache hits with <5ms.- Model serving fleet: - Low-latency pool: GPU/CPU instances optimized for inference (e.g., Triton/torchserve/TF-Serving) behind an internal LB. Keep model instances warm, use batch-size=1 or small micro-batching with latency-aware scheduler. Use pinned threads, mem-mapped models, and request coalescing. - Batch pool: separate cluster optimized for throughput (spot GPUs, large batch sizes, async workers, throughput-optimized scheduling). Runs scheduled jobs or streaming consumers reading from object storage / message queue.- Post-processing service: formats outputs, applies business rules, logs metadata.- Storage: object store for large inputs/outputs, feature store for offline features.- Autoscaler: two-tier autoscaling: - Online: predictive scaler using recent QPS and queue length + control loop to maintain 200ms p95; scale by adding warm instances (scale-out pre-warming). - Batch: job-based autoscaling reacting to queue depth and cost constraints (use spot/preemptible with checkpointing).- Orchestration: Kubernetes (node pools for gpu/cpu + taints/tolerations) or managed inference services.Key design choices & trade-offs:- Separation vs shared infra: - Separate pools (recommended): isolates SLOs, allows different instance types, scheduling, cost optimizations. Prevents batch spikes affecting online latency. Slightly higher operational overhead and resource duplication. - Shared pool: lower cost (better resource utilization) but requires strict QoS enforcement, prioritization, admission control and complex autoscaling to preserve latency SLOs—riskier for tight p95.- Caching: aggressive for repeated queries or idempotent responses to drastically reduce load on model servers. Cache TTL, invalidation tied to model/version changes.- Micro-batching: used in online only if latency budget allows; in batch use large batches for GPU efficiency.Monitoring & observability:- Metrics: - Latency histograms (p50/p95/p99) end-to-end and per-component. - QPS, instance utilization (CPU/GPU), queue lengths, batch sizes. - Cache hit ratio, pre/post-processing latency, error rates.- Tracing: distributed tracing (e.g., OpenTelemetry) capturing request path, model version, batch id to surface tail-latency causes.- Logging: structured logs with request IDs, input summary (hashed), model version, resource usage.- Alerts: - SLO breach alerts (p95 > 200ms), sustained high error rate, low cache hit rate, GPU OOMs. - Predictive alerts for scaling (forecasted QPS).- Canary & rollout: traffic split for new model versions; compare metrics (latency, correctness) before full rollout.- Cost & capacity dashboards: daily/weekly cost per workload, spot eviction rates for batch.Edge cases & resilience:- Graceful degradation: fall back to cached or simpler model when overloaded.- Circuit breakers & rate limiting per tenant.- Checkpoint/resume for batch on preemption.Summary: use separated, specialized clusters for online (latency-optimized, warm instances, predictive autoscale, cache) and batch (throughput-optimized, large batching, spot usage), with shared components for preprocessing and storage where sensible. Strong observability and admission controls protect the 200ms p95 SLO while keeping batch cost-effective.
Unlock Full Question Bank
Get access to hundreds of Architecture and Technical Trade Offs interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.