System Architecture and Tradeoffs Questions

Ability to decompose complex systems into components and define clear responsibilities, interfaces, and interactions. Evaluate architectural alternatives and articulate core trade offs such as consistency versus availability, latency versus throughput, simplicity versus extensibility, and cost versus performance. Explain how design choices affect scalability, resilience, failure modes, and operational burden, and justify architecture decisions based on expected load patterns and business requirements.

HardTechnical

55 practiced

Design an observability stack for a global microservice fleet producing petabytes of logs/month and very high-cardinality metrics. Propose sampling strategies, retention tiers, metric rollups, trace-sampling rules, and ways to retain debuggability while drastically reducing storage and query costs.

Sample Answer

**Situation & goals (brief)** I’d design an observability stack that preserves debuggability while cutting storage/query cost ~5–10x for a global microservice fleet producing PBs/month.

**High-level design**- Central multi-tenant pipeline: local agents → regional Kafka → processing layer (Vector/Fluentd) → cold/hot tiers in object store + indexer.- Use metadata catalog to map service/host/slo tags for selective retention.

**Sampling & ingest controls**- Ingest-level adaptive sampling per-service: preserve 100% error/latency/failed requests; reservoir/sample ratios for normal traffic (e.g., 1:1000) tuned by traffic type.- Rate-limits with fingerprinting to keep at-least-one-per-user/session.

**Metric strategy**- Separate high-cardinality labels into sparse-indexed dimensions; drop low-value labels at ingest.- Rollups: pre-aggregate to 1s/10s at hot tier, 1m/5m for medium, 1h for long-term.- Use bloom filters + sketches (t-digest, HDR histograms) for distribution metrics to keep percentile accuracy.

**Trace sampling**- Head-based sampling: keep all traces for errors/alerts; probabilistic for success traces (e.g., 0.1% baseline).- Priority sampling: boost traces for unusual spans or anomalous latency signatures; keep sampled span trees for context.- Link traces to sampled logs/metrics via stable trace IDs and sample-retention bookmarks.

**Retention tiers & storage**- Hot (7–30d): full logs, full-resolution metrics, traces for retained samples — SSD/TSDB.- Warm (30–90d): compressed logs (columnar/parquet), aggregated metrics (1m), sampled traces — object store with index shards.- Cold (90d+): compressed rollups, sketches, and sparse indexes for on-demand retrieval; cost-based restore.

**Maintaining debuggability**- Always keep: full traces for failures, top-N slow endpoints, and a per-user/session breadcrumb sample.- On-demand fetch: keep small index to efficiently restore raw logs/traces from cold storage for specific trace IDs.- Implement “forensic mode” to temporarily increase sampling for a service if an incident is declared.

**Trade-offs & governance**- Accepts lower fidelity for normal traffic; requires robust alerting to detect undersampling issues.- Provide SLO-driven default policies; allow teams overrides with quotas.- KPIs: storage $/month, query latency, incident MTTI/MTTR to validate.

This balances cost and signal retention while giving teams control and fast incident workflows.

HardTechnical

72 practiced

You oversee 20 engineering teams. Propose an architecture governance model that balances team autonomy with organization-level standards for security, scalability, and API contracts. Include review cadence, guardrails (automated + manual), an exceptions process, and KPIs to ensure governance works without becoming a bottleneck.

EasyTechnical

52 practiced

You're launching an internal service with little historical telemetry. Describe a pragmatic process to set SLOs and SLAs with product and support stakeholders: how to pick target latency/error budgets, how to instrument to measure them, and what actions to take when an error budget is burned.

Sample Answer

**Situation & goals**As Engineering Manager launching an internal service, my goal is to set pragmatic SLOs/SLA that balance risk, developer velocity, and stakeholder expectations (product/support).

**Process to pick targets**- Align with stakeholders: run a short workshop with product, support, and key users to capture critical user journeys and acceptable business impact.- Map intents to metrics: for each journey choose latency (p95/p99), availability (success rate), and business KPIs (e.g., time-to-ack for alerts).- Use conservative defaults with limited history: start with industry norms (e.g., 99.9% availability, p95 latency target) and translate to error budgets (1000ms breaches or 0.1% failures → budget = 0.1%).- Timebox review: commit to a 3-month observation window and iterate targets after real telemetry.

**Instrumentation & measurement**- Instrument at entry/exit of service and on important downstream calls; record timestamp, trace-id, status code, and user-journey tag.- Use distributed tracing (OpenTelemetry), metrics (Prometheus/Grafana), and alerting based on SLO burn rate.- Define dashboards: SLO burn rate, error budget remaining, top error types, and latency heatmaps.- Add synthetic tests and canaries to cover low-traffic paths.

**When error budget is burned**- Predefine escalation policy: warn at 50% burn, action at 80%, and freeze at 100%.- Actions by severity: - 50%: prioritize bug fixes, temporarily reduce noncritical releases. - 80%: stop non-essential launches, run incident review, dedicate a small rotating team to remediation. - 100%: rollback risky changes, schedule a postmortem, update runbooks, and re-negotiate SLA with product if needed.- Communicate: transparent status to stakeholders, expected remediation timeline, and compensating measures for affected teams.

**Outcome & learning**- Treat SLOs as living agreements: after the observation window, refine targets based on measured distribution, adjust error budgets for business impact, and record learnings into on-call/runbook.

MediumSystem Design

66 practiced

Design a low-latency leaderboard service for a competitive game with tens of thousands of score updates per second and global read-heavy traffic. Explain how you'd use caching, sorted data structures, sharding, update fanout, and trade-offs between accuracy and freshness.

MediumSystem Design

63 practiced

You manage a geo-distributed user-profile service for 100M users. Reads must be served from regional endpoints with <200ms latency; writes can be eventually consistent. Design replication, read/write routing, conflict resolution and failure modes for five regions and describe how you'd validate the architecture.

Sample Answer

**Clarify requirements & constraints**- 100M users, 5 regions, read latency <200ms from regional endpoints, writes eventually consistent, availability prioritized, conflict resolution acceptable.

**High-level architecture**- Active-active geo-replication: each region hosts a read replica and an acceptor for local writes.- Use a global control plane (config & metadata) and per-region data plane.- Per-region cache layer (edge CDN + regional Redis/Memcached) for hot profiles; backing store is multi-master database (e.g., DynamoDB global tables or Cassandra with CRDTs).

**Replication & routing**- Reads: route to nearest regional endpoint via geo-DNS / anycast + health checks; serve from regional cache, fall back to local DB.- Writes: sent to local region; writes are locally acknowledged and asynchronously replicated to other regions via change-data-capture (CDC) + ordered log (Kafka).- Use per-profile sharding by user-id to distribute load.

**Conflict resolution**- Prefer CRDTs (LWW-register with vector-clock metadata for complex fields) or per-field last-writer-wins with user-centric merge rules (e.g., merge arrays, authoritative timestamps for critical fields).- For high-sensitivity data (email, password): enforce a global sequence via lightweight leader election (per-shard leader) or synchronous compare-and-swap.

**Failure modes & handling**- Region outage: global routing shifts reads to next nearest; writes accepted in other regions. On recovery, CDC replay with idempotent application; resolve conflicts via vector clocks/merge policies.- Network partition: risk of conflicting writes — detect via vector clocks and surface to app if auto-merge ambiguous.- Backpressure: throttle non-critical writes, prioritize profile reads.

**Validation & testing**- SLOs/metrics: p95 read latency <200ms, replication lag, conflict rate, availability.- Tests: chaos engineering (regional failover, network partition), failback tests, synthetic traffic from each region for latency and correctness, consistency tests (write in region A, read in B after defined intervals), load tests at peak QPS.- Observability: dashboards for replication lag, per-shard errors, vector-clock anomalies, and automated alerts.

**Team & rollout**- Phased rollout: dev->staging->canary regions, runbook for leader-election and recovery, postmortems. Assign cross-regional ownership: replication team, conflict-resolution owners, and SRE-runbook maintainers.

Unlock Full Question Bank

Get access to hundreds of System Architecture and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.