System Design and Architecture Fundamentals Questions

Comprehensive coverage of designing scalable, reliable, and maintainable software systems, combining foundational concepts, common architectural patterns, decomposition techniques, infrastructure design, and operational considerations. Candidates should understand core principles such as horizontal and vertical scaling, caching strategies and placement, data storage trade offs between relational structured query language databases and non relational databases, application programming interface design, load distribution and fault tolerance. They should be familiar with architectural styles and patterns including client server and layered architectures, monolithic and microservices decomposition, service oriented and event driven designs, gateway and proxy patterns, and resilience patterns such as circuit breakers and asynchronous processing. Assessment includes the ability to decompose a problem into logical components and layers, define component responsibilities, map data flows between ingestion processing storage and serving layers, and select appropriate infrastructure elements such as application servers caches message queues and database replication models. Interviewers evaluate estimation of scale and load and reasoning about trade offs such as consistency versus availability and partition tolerance latency versus throughput coupling versus cohesion and cost versus complexity, and the ability to justify architecture decisions. Candidates should be able to sketch high level designs, communicate architecture to technical and non technical stakeholders, propose migration paths such as when to combine or transition between patterns, and describe operational runbooks including failure mode mitigation monitoring observability and incident recovery. Practical topics include caching eviction policies such as least recently used and least frequently used load balancing approaches such as round robin and least connections rate limiting techniques replication and sharding strategies and design choices for synchronous request response versus asynchronous queue based messaging. Emphasis is on clarifying requirements estimating constraints proposing reasonable architectures and articulating trade offs and evolution paths rather than only low level implementation details.

HardSystem Design

76 practiced

Design a solution to audit and attribute inference cost per customer for a shared multi-tenant LLM cluster where cost depends on tokens, model used, and GPU time. Include tagging, sampling approaches, high-accuracy vs low-cost accounting methods, and integration points with billing systems.

Sample Answer

Requirements & constraints:- Attribute cost per customer for shared multi-tenant LLM cluster where cost = tokens (in/out), model selection (cost-per-token varies), and GPU time (utilization, model inference time).- Low-latency accounting for billing integration, support high-accuracy audits, scalable to millions of requests, privacy isolation between tenants.

High-level architecture:- Ingress API / Request Router → Request Tagger → Accounting Collector (streaming) → Sampling Engine → Cost Calculator (two modes: High-accuracy & Low-cost) → Audit Store (+ cold raw logs) → Billing Connector.

Core components:1. Request Tagger (at ingress)- Attach immutable metadata: tenant_id, request_id, model_version, priority, tenancy_mode.- Enforce tags via JWT claims or signed headers; fail requests without valid tenant identity.2. Lightweight Instrumentation (edge)- Emit per-request events: tokens_in, tokens_out (estimated pre/post), model_name, start_time.- Send to streaming layer (Kafka/Kinesis).3. GPU Telemetry Agent- Per-GPU process-level mapping: maps process/pid/container → request_id(s). Uses cgroups + NCCL/profiler hooks to capture precise GPU time per request batch. Emits GPU_time samples.4. Sampling Engine- For high-throughput: sample a configurable fraction of requests (stratified by tenant size, model, latency bucket). Ensure every tenant has deterministic minimum sample rate.- For small tenants: always full-accounting (no sampling).5. Cost Calculator- Low-cost mode: estimate GPU_time = model_cost_per_token * tokens_total + empirically derived overhead per model; use token counts + model price to compute near-real-time costs.- High-accuracy mode: for sampled requests, use exact GPU_time telemetry + exact token counts. Train per-model regressors mapping tokens and metadata → GPU_time; apply to non-sampled requests to reduce bias.6. Audit Store & Reconciliation- Store raw events (compressed), sampled telemetry, and computed attributions. Support backfill and periodic reconciliation jobs that compare estimates to sampled ground truth; adjust regressors and bias corrections.7. Billing Integration- Expose per-tenant daily/hourly cost reports via push to billing system (invoices, usage records) and webhooks. Provide “tentative” (estimated) and “final” (audited after reconciliation) amounts. Support dispute API to fetch raw logs for a tenant's invoiced window.

Data flow:- Ingress tags → streaming → real-time low-cost estimator updates per-tenant counters (tokens, estimated GPU_time) → billing connector emits provisional records.- Sampled requests write GPU telemetry → high-accuracy pipeline computes precise cost, updates audit store, triggers reconciliation which adjusts historical tenant billing as needed.

Scalability & accuracy trade-offs:- Low-cost estimator: O(1) per request, minimal overhead, suitable for large scale but biased.- High-accuracy: expensive telemetry and storage; limited to samples and periodic audits.- Use stratified sampling to reduce variance for mid-size tenants; enforce minimum sample per tenant per day.

Security & compliance:- Tenant separation via tags and auth; encryption in transit/storage; retention policies for raw logs; allow tenants to opt-in/out of detailed telemetry with contractual constraints.

Operational considerations:- Monitoring of sampling bias, drift in regressors, and model upgrades (retrain regressors per model_version).- Rate-limit telemetry to avoid perf impact; fallback to token-only accounting if telemetry unavailable.- Provide dashboards for finance and tenant-facing usage/cost breakdown and a reconciliation SLA.

Trade-offs:- More sampling increases accuracy but more GPU/time instrumentation cost.- Conservative billing: charge estimated amount and credit final audited difference to avoid overcharging.

This design balances real-time low-cost accounting for billing operations with sampled high-accuracy auditing to correct systemic bias and support dispute resolution.

MediumSystem Design

72 practiced

Design an architecture for a real-time conversational platform where conversation state must be consistent across devices and available in multiple regions. Requirements: 100M monthly active users, 10k RPS global, per-conversation state durability. Discuss state storage patterns, session affinity, and replication strategies.

Sample Answer

Requirements & constraints (clarify):- 100M MAU, ~10k RPS global, per-conversation durable state, multi-region low latency, strong-ish consistency across devices.

High-level architecture:Client ↔ Regional Edge API Gateways (TLS, auth, rate-limit) ↔ Regional Frontend / Session Router ↔ Conversation State Layer + Message Broker ↔ AI/Inference services & Persistence

State storage patterns:- Hot-state: per-conversation session cache in regional in-memory stores (Redis Cluster with AOF/RDB) for low-latency read/write of active conversations.- Durable-state: authoritative per-conversation records in a multi-region primary/secondary DB (Cloud Spanner, CockroachDB, Cosmos DB) or a write-local + async replicate pattern to avoid cross-region sync on every message.- Append-only message log: store every message/event in an immutable, partitioned object store (Kafka + Tiered storage or CDC → S3) for replay, auditing, and rebuilding state.

Session affinity:- Use sticky routing at the edge: route client to nearest region. Within region, use consistent-hash (conversation_id) to pick a frontend instance and the associated Redis shard. This minimizes cross-shard reads and keeps latency low.- Short-lived tokens and session metadata kept in JWT + Redis to allow seamless failover between devices.

Replication strategies & consistency:- For active-session consistency across devices, use a hybrid model: - Synchronous local-write, asynchronous cross-region replicate: write to regional Redis + append to local Kafka; local DB write is committed synchronously. Cross-region replication via Kafka MirrorMaker or geo-replication pipelines ensures eventual consistency with low write latency. - For conversations requiring strong cross-device consistency (e.g., payments, bookings), use a distributed transactional DB with linearizable writes (Spanner/Cockroach) for those keys.- Convergence: use operation logs with causal metadata. For conflict resolution across regions prefer CRDTs (for merges like typing presence, cursors) or last-writer-wins with vector clocks for messages when business logic tolerates it.

Durability & recovery:- Persist checkpoints from Redis to durable DB frequently (AOF or snapshot + async flush). Rehydrate hot-cache from append-only logs on failover.- Use CDC to populate read-replicas across regions; maintain tombstone/deletion policies.

Scalability & availability:- Partition conversations by conversation_id across clusters; autoscale frontends and Redis shards.- Rate-limit and throttle heavy inference; use batching and async responses for model calls.- Multi-region failover: health checks + DNS/anycast to redirect traffic; promote secondary DB region if primary unavailable.

Trade-offs:- Synchronous cross-region writes give strong consistency but hurt latency and throughput.- Async replication + CRDTs favors availability and low latency but requires careful conflict resolution.

Metrics to monitor:- write/read latency, replication lag, cache hit rate, message loss, RPO/RTO targets.

This design balances per-conversation durability, low regional latency, and global availability by combining fast regional caching, append-only logs, and selective transactional storage for strong-consistency paths.

MediumTechnical

112 practiced

Design a cost-optimized deployment for a low-usage but latency-sensitive inference endpoint with sporadic traffic. Evaluate serverless vs pinned instances vs warm-pool approaches, discussing cold-start impact and cost/performance trade-offs.

Sample Answer

Requirements & constraints:- Very low QPS but latency-sensitive (p95 target e.g. <200ms).- Sporadic traffic with unpredictable spikes.- Cost needs minimizing (idle costs matter).- Model may require CPU or GPU; assume medium-size transformer that benefits from GPU but can be quantized to CPU.

Options evaluated

1) Fully serverless (FaaS / cloud serverless inference)- Pros: Pay-per-invocation, zero idle infra cost, easy autoscaling.- Cons: Cold-start latency (container startup + model load) can be hundreds of ms–seconds, unpredictable; limited GPU availability or high cost; limits on memory/ephemeral storage.- Best if model is small and can be kept in-memory quickly (or if p95 latency target allows small cold-starts).

2) Pinned instances (always-on VM/GPU)- Pros: Lowest tail latency; model stays resident, no cold-starts; predictable performance.- Cons: High base cost — wasteful for very low utilization, especially GPUs.

3) Warm-pool / hybrid (small pool of pre-warmed containers/instances + scale-to-zero)- Pros: Balances cost and latency: small number of pre-warmed replicas avoid cold-starts for most traffic; autoscale for bursts; can scale to zero overnight except warm-pool.- Cons: Some idle cost for warm pool; added operational complexity.

Cold-start impact & mitigation- Cold-start components: container VM provisioning, runtime init, model loading. For large models, model load dominates.- Mitigations: model quantization/pruning, lazy-loading only necessary parts, snapshotting warmed process (e.g., keep a checkpointed memory image or use fast resume), smaller distilled models for initial responses, use compiled runtimes (TorchServe, ONNX Runtime).- For serverless, use “provisioned concurrency” (AWS Lambda) or warmers; for k8s, use HPA + custom controller to maintain N warm pods.

Cost/performance trade-offs & recommendation- If model can be CPU-quantized without violating accuracy and cold-start must be <200ms: use serverless with provisioned concurrency sized to expected burst capacity (keeps latency, moderate cost).- If model needs GPU for acceptable latency/accuracy: use a tiny warm-pool of GPU instances (1 small GPU instance or 2 fractional GPUs if supported) plus rapid autoscaling to on-demand GPU instances for spikes. Combine with a CPU-based distilled fallback model for ultra-low latency when GPU cold.- For strict cost minimization with acceptable occasional cold starts: serverless on cold path + warm-pool of 1 pre-warmed instance during business hours.

Operational plan (concrete)- Measure actual traffic pattern -> choose warm-pool size = p95 concurrent requests during steady-state.- Implement autoscaler with custom metrics (inflight requests, queue latency) and cooldown windows.- Use model optimizations: quantize, use ONNX + vectorized batching, enable model caching in memory.- Add health-check based warmers and graceful request queuing with short timeouts and fallback distilled model.

Example numbers (illustrative)- Always-on GPU: $0.9–3/hr => ~$648–2160/mo.- Warm-pool of 1 GPU used part-time (e.g., 10 hrs/day): ~ $270–900/mo.- Serverless provisioned concurrency: depends on provider; often cheaper at very low throughput but can be expensive if provisioned high.

ConclusionFor an AI engineer: prefer a hybrid warm-pool approach—keep a minimal number of pre-warmed replicas (GPU or optimized CPU) to meet p95 latency, autoscale for spikes, and use model compression + CPU fallback to reduce required warm capacity. This yields predictable latency while minimizing idle cost.

HardSystem Design

73 practiced

Architect a solution that supports live migration of stateful conversational sessions between data centers to reduce latency for users who move regions. Ensure no message loss and minimal disruption. Discuss session transfer mechanics, routing updates, locks, and consistency guarantees.

Sample Answer

Requirements:- Live migrate stateful conversational sessions across regions with <200ms disruption, zero message loss, preserve ordering, support thousands of concurrent migrations, secure transfer.

High-level architecture:- Session Store (primary in source DC), Replicated Append-Only Log (cross-region durable), Control Plane service (migration orchestrator), Proxy/Routing layer (regional load balancers + session-aware router), Client sticky token (session id + migration epoch), Locking service (distributed lease), and short-lived Transfer Worker per session.

Session transfer mechanics:1. Begin: client signals movement (or orchestrator detects via latency). Orchestrator chooses target DC and initiates Transfer Worker.2. Freeze & checkpoint: Acquire lease on session (short optimistic lock via etcd/consensus). Pause new user input routing to worker by setting session state = "migrating" in control plane.3. Drain & log: Source DC continues to accept in-flight messages but appends them to the replicated WAL and buffers outgoing responses. Once drain window closed (few hundred ms), write a final checkpoint snapshot to object store and persist a migration marker with sequence number SN_end.4. Replicate: WAL and snapshot are replicated to target DC via durable replication (RAFT/CDC). Target replays WAL up to SN_end ensuring identical state and message ordering.5. Cutover: Target acquires lease, marks session active, updates control plane routing (atomic metadata update with migration epoch). Routing change is propagated to proxies (push config) and client receives new endpoint or uses existing anycast with session token that encodes epoch.6. Resume: Target starts accepting new messages; any late-arriving messages to source are forwarded (proxy-level) to target using sequence numbers and idempotent apply.

Routing updates:- Use control plane metadata: session_id -> (region, epoch, lease_expiry).- Proxies consult control plane cache; propagation via pub/sub (low-latency).- Client-side token with epoch allows direct routing or validation.- For DNS-based clients, use short TTLs and connection handoff via WebSocket/TCP-level proxy that tunnels to target during transition.

Locks and concurrency:- Short-lived leases prevent split-brain. All writes must include sequence numbers and idempotency keys.- During migration, source accepts but forwards or appends to WAL; writes rejected only if lease expired.- Two-phase handoff: source drains and forwards, target only marks active after full replay and lease acquisition.

Consistency guarantees:- Strong consistency for ordering via WAL with global sequence numbers per session.- Exactly-once semantics via idempotency keys and dedup table per session.- No-loss ensured by durable WAL replication and final checkpoint acknowledgment before cutover.

Operational considerations & trade-offs:- Latency spike vs. durability: short drain window balances user interruption and consistent snapshot.- Heavy cross-region replication costs — optimize by sharding WAL and compressing deltas, transfer only deltas since last sync.- Failure modes: if target fails during replay, orchestrator rolls back to source lease; if source fails mid-transfer, replicated WAL + snapshot allow target to complete.- Monitoring: migration metrics, latencies, replication lag, lease status, and automated rollback thresholds.

This design gives ordered, durable, minimal-disruption live migration via lease-protected handoff, WAL-based state transfer, idempotent application of messages, and atomic routing updates.

MediumTechnical

87 practiced

Design an embedding cache strategy that avoids serving stale vectors after a model update. Discuss invalidation, versioned keys, migration during rollout, and approaches for minimizing cache misses and cold starts.

Sample Answer

Situation / Requirements (brief): We must ensure embeddings served to downstream retrieval/inference are never stale after a model update, while keeping performance high (low latency, few cache misses) and supporting rolling upgrades.

Design summary:1. Versioned keys- Store embeddings with a version suffix: embed:{model-version}:{id}- Model version advances on any embedding-significant change (architecture, training, tokenizer)- A global model-version metadata object in a strongly-consistent store (e.g., Redis, etcd, DB) indicates current active version.

2. Read path (avoid stale vectors)- On request, read current model-version from metadata (cached locally with short TTL, or read-on-change via pub/sub).- Lookup embed:{version}:{id}. If hit, return.- If miss, compute embedding with current model, write embed:{version}:{id} and return.

3. Invalidation & migration during rollout- Blue/Green or Canary: deploy new model version alongside old. The metadata switch only after canary passes.- Background migration: launch workers that scan dataset and precompute new-version embeddings into cache (bulk-warm).- On metadata switch, stop computing old-version embeddings for new requests. Old keys kept for fallback for a TTL or removed via GC.

4. Minimizing cache misses & cold starts- Proactive warm-up: upon training completion, run batched embedding jobs (high-priority) for top-K frequent items and for index shards.- Lazy but prioritized warm-up: on first miss, compute and async replicate to cache; meanwhile serve from fallback (old-version embedding) if soft-consistency acceptable, or block briefly for strong-consistency.- Hot-key handling: maintain access counters; pre-warm hot keys on version switch.- Use sharded background workers + rate-limited re-embedding to avoid compute spikes.

5. Consistency trade-offs & throttling- Strong consistency: block until new embedding computed (adds latency).- Eventual consistency: serve old embedding until new is available (lower latency, short-lived staleness).- Choose per endpoint based on SLA; critical endpoints use strong-consistency.

6. Operational details- Pub/sub: broadcast version-change so caches invalidate local in-memory copies instantly.- GC policy: TTLs per version; expire old-version keys after rollback window.- Metrics & alarms: cache hit/miss by version, migration progress, re-embed throughput.- Testing: simulate rollback and partial failure to validate safe fallback.

Why this works:- Versioned keys eliminate ambiguity and prevent silent staleness.- Background and prioritized warm-up minimize cold-starts.- Controlled metadata switch + pub/sub enables atomic rollout with observability and rollback support.

Unlock Full Question Bank

Get access to hundreds of System Design and Architecture Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.