System Design Problem Solving and Methodology Questions

A structured approach to solving open ended system design problems during interviews. Emphasis on requirement gathering and clarifying questions, making and stating assumptions explicit, calculating capacity and load estimates, identifying and prioritizing bottlenecks, proposing modular and testable solutions, and articulating trade offs with respect to performance cost reliability and time to implement. Also covers communication of ideas using diagrams, incremental delivery and backward compatible changes, and how to justify design decisions under uncertainty.

EasyTechnical

72 practiced

Explain the CAP theorem in practical terms and give one concise example where you would choose availability over consistency and another example where you would choose consistency over availability. For each example, state the business requirement driving the choice and the operational consequence.

MediumTechnical

61 practiced

Design a rate-limiting system for an API gateway that must enforce per-API-key and per-IP limits. Compare algorithms (token bucket, leaky bucket, fixed window), discuss central vs local enforcement, how to synchronize state across gateway instances, cold-start behavior, and how to allow controlled bursts without penalizing legitimate traffic.

Sample Answer

Requirements (clarify): enforce per-API-key and per-IP limits, allow controlled bursts, low latency at gateway, scale to many gateway instances, resilient to instance failure, reasonably strict accuracy (soft or hard limits?), and cost constraints.

Algorithms — pros/cons:- Fixed window: simple, cheap (increment per window). Problems: boundary spikes, poor burst control, coarse accuracy.- Sliding window / fixed window with sliding-log: better accuracy but higher storage/compute (per-request timestamps).- Leaky bucket: enforces constant outflow (good smoothing), simple to reason about; can reject bursts aggressively.- Token bucket: preferred for API throttling — tokens refill at rate R and allow bursts up to capacity B. Allows controlled bursts for legitimate traffic while enforcing average rate.

Central vs local enforcement:- Centralized (single Redis/DB): strong consistency, single source of truth, easier enforcement and policy changes; single point of latency/failure — mitigate with HA and local caching.- Local (per-instance token buckets): lowest latency, no single point, but requires synchronization to avoid global overuse and has cold-start/imbalance issues.

Recommended hybrid:- Primary enforcement locally using token-bucket per key/IP with conservative local capacity (e.g., local token reserve = B_local). Back each key/IP with a central store (Redis or DynamoDB) holding global token counts or last-refill timestamps.- On each request: attempt local token; if exhausted, do a fast async check-or-reserve to central store (or fall through to centralized rate check) to avoid blocking common case.- Use approx. distributed counters with Redis INCR + TTL or Lua scripts for atomic updates for strict limits.

Synchronizing state:- Use central Redis cluster with atomic scripts for updates. Store last_refill_ts and tokens for each key/IP. Gateways periodically pull or reconcile diffs (push on change) and use optimistic local consumption.- For scale, shard keys across Redis shards, use consistent hashing, and cache permissive tokens locally with TTL.

Cold-start behavior:- New gateway instances get a small local token bucket (grace) and fetch state from central store on first traffic. Use rate-limited backoff to avoid thundering herd. Initialize local capacity conservatively.

Allowing controlled bursts:- Use token-bucket capacity tuned per SLA. Different classes: normal (B), premium (2B). Implement burst windows: allow larger short-term bursts but enforce longer-term average via refill rate.- Smooth bursts by prioritizing user-visible traffic, queue low-priority requests briefly (small buffer) rather than immediate hard reject, and use 429 with Retry-After.

Operational considerations:- Metrics/alerts for global saturation, per-key abuse detection, and counters exported to monitoring.- Testing: chaos inject Redis latency/failures and validate fallback behaviors.Trade-offs: stronger consistency increases latency and operational complexity; hybrid gives best latency/UX while keeping global control.

HardTechnical

52 practiced

Design an immutable audit logging architecture to meet strict regulatory requirements: every configuration change and user action must be immutably logged, searchable, tamper-evident, and retained for policy periods. Discuss append-only storage, indexing for search, encryption, access controls, retention/archival, and the performance trade-offs.

Sample Answer

Requirements (clarify): immutable, tamper-evident audit trail of config changes and user actions; fast searchable queries (by user, resource, time, event type); encryption at-rest/in-transit; strict RBAC + separation of duties; retention and legal hold/archival; scalability and acceptable write/read latency for production.

High-level architecture:- Clients (apps, infra, admin consoles) -> authenticated, signed events -> Ingest layer (API gateway, Kafka or Kinesis).- Append-only storage: WORM-backed object store (S3 Object Lock/Governance or AWS S3 Glacier Vault Lock) or write-once ledger DB (DynamoDB with immutable items + Change Record Table) with transaction log persisted to S3. Each event packaged with metadata, event hash, sequence number, and signer certificate.- Tamper-evidence: Periodic Merkle tree roots computed over ordered blocks; root published to external anchor (blockchain, separate hardened timestamping service, or cross-account S3). Store per-block signed manifests.- Indexing/search: Stream processors (AWS Lambda/Glue/Flink) consume ingest stream to build read-only indices in a search cluster (Elasticsearch/Opensearch or vector DB) that holds only indexed metadata (not raw events) and immutable pointers to raw objects. Full-text indexing limited to allowed fields to reduce leakage.- Encryption: Envelope encryption — each event encrypted at-rest with data key, KEK in HSM/KMS. Signatures use keys in HSM. In-transit TLS + mTLS.- Access controls: Fine-grained RBAC and ABAC enforced at API/gateway and search layer. Read operations require justification/approval; write-only ingest endpoints; separation of duties: ops cannot delete/modify WORM objects; logs view via audited queries only.- Retention/archival: Lifecycle policies move objects to cold vaults (Glacier Deep Archive) after index TTL; retention enforced via WORM; support legal hold by tagging and preventing lifecycle changes. Periodic integrity checks (recompute Merkle root, validate signatures).- Audit and monitoring: Immutable audit of access to logs (who queried, when) written back into same system. Alerting on integrity failures.

Performance trade-offs:- Append-only WORM + Merkle anchoring adds write overhead (hashing, signatures) and increases latency vs plain logging; mitigate with batched blocks (e.g., 1–5s or size-bound) to amortize cost.- Indexing for search introduces eventual consistency — near-real-time (seconds) vs synchronous. If synchronous search is required, index writes add higher write latency.- Storage cost vs retrieval speed: keeping raw events on hot storage accelerates forensic queries but raises cost; recommended: keep hot recent window (30–90 days) then cold archive.- Search cluster must be read-optimized and isolated from write path to preserve immutability; scaling for ad-hoc forensic queries may require query concurrency limits or query-offloading to analytics jobs.

Trade-offs & choices:- Use managed services (S3 Object Lock + KMS + Lambda + OpenSearch) to reduce ops and meet compliance quicker; if requirement mandates independent proofs, anchor Merkle roots to public blockchain or external notary.- For highest assurance, deploy HSM-backed signing and multi-party attestation for root publishing.

Operational considerations:- Key rotation and re-encryption strategy that preserves immutability (store old keys for verification; do not re-encrypt existing WORM objects in place).- Regular compliance drills: restore tests, integrity verification, and retention/hold audits.- Documentation for legal and auditors: data model, retention policy, cryptographic proofs, and access logs.

This design balances strict immutability and tamper-evidence with searchability and cost by separating append-only raw storage from indexed search, using batched cryptographic anchoring and managed security primitives to meet regulatory requirements.

MediumTechnical

71 practiced

When designing a system for scale, provide a concrete checklist you would use to identify and prioritize bottlenecks. Include how you diagnose CPU, network, storage, contention, database hotspots, third-party dependency limits, and how to decide between immediate mitigations (caching, retries) versus architectural changes (sharding, redesign).

Sample Answer

Checklist for identifying & prioritizing bottlenecks (for scale)

1) Clarify objectives & constraints- SLA, latency p95/p99 targets, QPS, data retention, cost, deployment window.

2) Gather baseline metrics & traces- Collect metrics (CPU, mem, disk I/O, network, request latency), distributed traces, DB slow queries, contention stats, and dependency SLAs for a representative load.

3) Reproduce and rank hotspots- Run controlled load tests and canary in prod. Rank by user-impact = (frequency × severity × business cost).

Diagnosis checklist (concrete checks)- CPU: per-process CPU %, sys vs user time, run-queue length, profiling (flame graphs) to find hot functions.- Memory: OOM, GC pause times, heap/stack growth patterns.- Network: interface utilization, packet drops/retries, latency histogram, TCP retransmissions, MTU misconfig.- Storage: IOPS vs latency, queue depth, fsync frequency, read/write amplification, SSD vs HDD characteristics.- Contention: lock contention, thread pool saturation, queue lengths, mutex/DB row locks via profiling/traces.- DB hotspots: slow queries, index misses, lock waits, hotspot keys (use query plan, histograms, shard skew).- Third-party limits: rate limits, throttling headers, downstream latencies/error rates, SLA breach risks.

Mitigation decision framework- Short-term (fast, low-risk): caching (CDN, in-memory), connection pooling, retries with backoff, circuit breakers, read replicas, query tuning. Use when symptom isolation shows quick wins and risk tolerable.- Medium-term: horizontal scaling (stateless replication), async decoupling (message queues), batching, optimized serialization.- Long-term (architectural): sharding/partitioning, redesign data model, CQRS, re-architect monolith -> services. Choose when bottleneck is fundamental (single-point hotspot, cardinality limits) or short-term fixes hit cost/complexity ceiling.

Cost-risk-priority matrix- For each candidate fix evaluate: latency improvement, dev effort, deployment risk, operational cost. Prioritize high-impact/low-effort first; reserve architectural changes for persistent or growing constraints.

Validation & monitoring- After changes, re-run load tests, confirm p95/p99 improvements, add dashboards and alerts, and document runbooks.

MediumTechnical

62 practiced

As a solutions architect during a sales cycle, outline a proof-of-concept (PoC) plan that validates technical feasibility without a full implementation. Define scope (what to build), success criteria, minimal architecture, required test data, metrics to measure, timeline, stakeholder responsibilities, and rollback/hand-off strategy to engineering.

Sample Answer

Scope (what to build)- A thin, working slice of the proposed solution that exercises the highest-risk capabilities: authentication/authorization, core data flow, a representative integration (e.g., upstream API or database), and an end-to-end happy-path UI or API. No nonessential features (analytics, multi-region, heavy optimization).

Success criteria (pass/fail)- Functional: end-to-end transaction completes with expected data persisted and returned for 95% of test cases.- Non-functional: average API latency < 300ms under validation load; memory/CPU within 20% of target production estimates.- Integrations: third-party system accepted and returns expected schema; security scan shows no critical findings.- Business: stakeholder acceptance sign-off and a documented list of remaining gaps (<=5 known gaps).

Minimal architecture- Lightweight PoC stack: single app service (container), a managed DB (or test schema), a stub/mock service for any low-priority external system, and a small API gateway or load balancer. Instrumentation: tracing, basic metrics exporter, and logs shipped to a temporary observability workspace.

Required test data- 3–5 representative datasets covering normal, boundary, and error cases (e.g., small/large payloads, missing fields). Synthetic user accounts with three roles. Mocked external responses including latency/error scenarios.

Metrics to measure- Functional success rate (% of successful end-to-end flows)- Latency (p50/p95/p99)- Throughput (transactions/sec under test)- Error rate and exception counts- Resource usage (CPU, memory)- Time-to-recovery for an induced fault

Timeline (4 weeks example)- Week 0: Align scope, success criteria, and access; prepare environment- Week 1: Build minimal architecture and wire happy path- Week 2: Add integrations, test data, instrumentation- Week 3: Run validation tests (functional + load), capture metrics- Week 4: Review results, stakeholder demo, produce hand-off doc

Stakeholder responsibilities- Solutions Architect: define scope, design, lead PoC, run demos- Sales/Account Exec: business objectives, stakeholder availability, sign-off- Customer SME/IT: provide access, test data, validate business cases- Dev/Integrator (vendor or customer): implement minimal components, resolve blockers- Security/Compliance: run quick review and accept residual risks

Rollback / hand-off strategy- Keep PoC isolated (separate infra, tags, and resource naming). Document full runbook: deployment steps, infra-as-code, data teardown, and known limitations. For rollback: scripted destroy of PoC resources and data retention policy. For hand-off: deliver architecture diagram, code repo with README, test data, measured metrics, issues backlog, and prioritized next steps for engineering to convert PoC into production-ready design. Include recommended non-functional enhancements and an estimated effort for production hardening.

Unlock Full Question Bank

Get access to hundreds of System Design Problem Solving and Methodology interview questions and detailed answers.

Join thousands of developers preparing for their dream job.