Architecture and Technical Trade Offs Questions

Centers on system and solution design decisions and the trade offs inherent in architecture choices. Candidates should be able to identify alternatives, clarify constraints such as scale cost and team capability, and articulate trade offs like consistency versus availability, latency versus throughput, simplicity versus extensibility, monolith versus microservices, synchronous versus asynchronous patterns, database selection, caching strategies, and operational complexity. This topic covers methods for quantifying or qualitatively evaluating impacts, prototyping and measuring performance, planning incremental migrations, documenting decisions, and proposing mitigation and monitoring plans to manage risk and maintainability.

HardTechnical

35 practiced

Construct a plan to detect and mitigate cascading failures where a downstream dependency's queue backlog causes synchronous upstream services to block and exhaust thread pools. Include changes at runtime, short-term mitigations, and longer-term architectural patterns to prevent recurrence.

Sample Answer

Situation: A synchronous upstream service blocks because a downstream dependency's queue backlog grows, exhausting thread pools and causing cascading failures. Below is an SRE-ready plan covering detection at runtime, short-term mitigations during an incident, and longer-term architectural fixes to prevent recurrence.

Detection & runtime observability- Metrics to add/monitor: downstream queue length, queue ingress/egress rates, service thread-pool utilization, request latency percentiles, active requests per instance, error rates, retry counts, downstream response time, circuit-breaker state.- Alerts: threshold alerts for queue length growth rate, thread-pool saturation (>80–90%), p99 latency spikes, and surge in queued requests. Add anomaly detection (rolling baseline) to catch unusual growth.- Dashboards & traces: link distributed traces showing synchronous call paths, heatmap of which callers cause most backlog, and per-instance thread-pool view. Enable request sampling at higher rate during incidents.

Short-term mitigations (incident runbook)- Immediate traffic control: - Enable upstream rate limiting / global ingress throttling (API gateway) to reduce pressure. - Deploy temporary reject/302/503 for non-essential traffic (feature flags).- Shed load gracefully: - Configure callers to fail fast: reduce client-side timeouts and lower max concurrent requests. - Enable circuit breakers (open if downstream errors/latency exceed thresholds) to stop recruiting threads.- Free resources: - Increase thread-pool capacity only if CPU/memory allow and backlog bounded; prefer conservative scaling. - If using containers/auto-scaling, scale up replica count and ensure cooldowns short.- Stop harmful retries: - Disable exponential retries or add jitter; convert immediate retries to asynchronous retry queue.- Stabilize queues: - If downstream can be paused, throttle ingress to its queue or temporarily increase its worker pool.- Communication & runbook steps: - Post incident channel, annotate metrics, and trigger postmortem if SLO impacted.

Longer-term architectural patterns- Asynchronous boundary & backpressure: - Convert synchronous critical paths to async where possible. Use durable queues with explicit backpressure signals and queue-length propagation. - Implement token-bucket/credit-based backpressure between services so upstream will stop sending when downstream is overwhelmed.- Bulkheads & isolation: - Use separate thread pools and connection pools per downstream dependency (bulkhead pattern) so a slow downstream doesn’t exhaust global threads. - Per-caller quotas to prevent noisy neighbors.- Circuit breakers & adaptive throttling: - Standardize circuit-breaker library with consistent thresholds and automatic recovery; tie it into service mesh or gateway for centralized control. - Implement adaptive rate limiting that reduces throughput proportional to observed downstream capacity.- Graceful degradation & prioritized traffic: - Define essential vs optional requests; shed low priority work first. - Implement feature flags to disable non-critical features under load.- Observability & testing: - Emit queue-level SLOs for downstream components and include in dashboards. - Add chaos tests that simulate queue backlogs and verify upstream behavior (chaos engineering).- Retry and idempotency: - Design retries to be idempotent and move to async retry queues with backoff.- Capacity planning & SLIs: - Model queueing (Little’s Law) to set capacity and worker counts; include queue-length SLOs and error budget policies that trigger protective actions.- Automation: - Automate mitigation playbooks: gateway throttles, open circuit breakers, scale commands.

Why these choices- Detection catches problems early; short-term mitigations reduce blast radius without risky code changes; long-term architecture prevents recurrence by removing synchronous coupling, isolating failures, and ensuring systems can signal and adapt to capacity constraints.

Example quick config changes (conceptual):- Gateway rate limit to 50% of normal- Client timeout = 500ms, retries = 0- Circuit breaker: open if p95 downstream latency > 2s for 30s

This plan balances immediate stabilizing actions with durable engineering changes to prevent future cascading failures.

EasyTechnical

28 practiced

Compare blue-green, rolling, and canary deployment strategies for an SRE team managing a service with strict availability SLOs. For each approach, list the operational tools required, rollback procedures, and how you would validate a canary before promoting it to full production.

Sample Answer

Blue-green, rolling, and canary each trade off risk, speed, and operational complexity. Below is a concise SRE-focused comparison with required tools, rollback steps, and how to validate a canary.

Blue‑Green- What: Two identical environments (Blue = current, Green = new). Switch traffic atomically.- Tools: CI/CD (Jenkins/GitHub Actions), infra-as-code, load balancer / DNS (ALB, Cloudfront, Route53), Kubernetes namespaces or separate clusters, monitoring (Prometheus/Grafana), synthetic tests.- Rollback: Re-point load balancer/DNS back to Blue (instant). If using DNS, ensure low TTL. Rollback is fast and low-risk.- Notes: Requires double capacity; good for strict SLOs when switch must be atomic.

Rolling- What: Replace pods/hosts gradually across the fleet (batch by batch).- Tools: Kubernetes Deployments/DaemonSets, orchestration (K8s, Terraform), CI/CD, health probes, automated canary hooks, monitoring.- Rollback: Stop rollout and trigger revision rollback (k8s rollout undo) which re-deploys prior replica set. Rollback speed depends on batch size.- Notes: Lower capacity overhead than blue-green but exposes old+new simultaneously.

Canary- What: Deploy new version to small subset of users/instances and progressively increase traffic after validation.- Tools: K8s + Istio/Linkerd for traffic routing, feature flags, CI/CD, Prometheus/Grafana, Alertmanager, distributed tracing (Jaeger), log aggregation (ELK/Datadog), automated canary analysis (Kayenta, Flagger, Argo Rollouts), synthetic test harness.- Rollback: Automated halt on threshold breach and route traffic back to stable; trigger CI/CD rollback or scale down canary group. Playbook should include rollback triggers and runbook steps.- Notes: Best balance of safety and resource use; suitable where fine-grained validation matters.

Validating a canary before promotion1. Define success criteria up-front tied to SLOs: error rate, p50/p95 latency, 5xx rate, saturation (CPU/memory), throughput, business KPIs.2. Duration & traffic ramp: run at each traffic step for a minimum observation window (e.g., 10–30 minutes) and stable sample size; ramp 1% → 5% → 20% → 50%.3. Automated checks: - Health checks and readiness probes - Synthetic end-to-end tests (critical flows) - Real-user metrics comparison vs baseline (statistical significance, e.g., 95% CI) - Tracing sampling to compare latency tails and error traces - Log error spike detection and SLO burn rate monitoring4. Automated canary analysis: use tools to compute score against thresholds and block promotion if degraded.5. Human guardrail: require on-call approval for final promotion when impacting critical SLOs.6. Post-promotion: keep elevated monitoring and a short warm rollback window.

Summary: For strict availability SLOs prefer blue-green when capacity allows for fastest safe rollback; use canaries + automated analysis for low-risk progressive validation when capacity or user segmentation enables it; use rolling for lower resource cost but ensure conservative batch sizes, probes, and quick rollback automation.

MediumTechnical

30 practiced

An SRE team must design a data retention and archival policy for traces and logs balancing cost and debuggability. Propose tiers, retention durations, sampling strategies, and fast-path access for recent incidents while keeping long-term forensic ability. Explain trade offs.

Sample Answer

Requirements & goals:- Preserve full-fidelity traces/logs for recent incidents (fast debugging).- Keep lower-cost long-term ability to investigate trends/forensics.- Minimize storage/query cost while preserving SLO-driven observability.

Proposed multi-tier policy

1) Hot tier (0–30 days)- What: Full-resolution traces + full logs (all spans, logs, tags).- Storage: Fast indexable store (Elasticsearch/Opensearch hot nodes, ClickHouse, or tracing DB).- Access: Fast-path: incident commander UI + direct query; preserve trace->log links; TTL 30d.- Use: Live debugging, RCA, rollbacks.- Trade-off: Highest cost but small window.

2) Warm tier (31–365 days)- What: Keep full traces for all errors/exceptions and sampled traces for successful requests (e.g., 1–5% head-based sampling). Store log metadata/indexes (timestamps, trace IDs, error tags) and truncated log bodies (first/last N KB).- Sampling strategy: Tail-based sampling to retain all traces with X% error rate or latency > p95, plus rate-limited retention of normal traces. For logs, keep full bodies for error logs, truncated for info/debug.- Storage: Cheaper nodes, compressed columnar store, or lower-tier indices.- Access: Slightly higher latency (~seconds–minutes) but queryable for trends and incident follow-ups.- Trade-off: Balance between ability to reproduce incidents vs storage cost.

3) Cold tier (1–7 years)- What: Aggregates, histograms, span summaries, search indices mapping trace IDs → archive location. Store only full traces for sampled error classes (e.g., 100 examples per error type per month).- Storage: Object store (S3), Glacier for older >1 year; use compressed protobuf/JSON.- Access: Rehydration on demand (hours to days) or query via precomputed rollups.- Use: Legal/compliance audits, long-term trend analysis.- Trade-off: High latency to retrieve full context.

Sampling & retention strategies- Hybrid sampling: combine head-based for steady sampling + tail-based triggers (error, latency, customer complaints). Example: keep 5% random head-sample of all traces; always keep traces where span error flag set or latency > p99.- Rate-limited retention: cap full error trace retention per service (e.g., 10k full error traces/day) to control burst costs; drop duplicates via signatures.- Log truncation & indexing: keep structured log fields and first/last 8KB of message; store payloads in object store if needed.- Metadata retention: Always keep compact index (traceID, timestamp, service, error codes, userIDs) for the full archival period to enable targeted rehydration.

Fast-path access for incidents- Short-term cache: maintain a rolling cache (hot tier) for last 30 days with fast query and tracing UI.- Incident enrichment: when an alert fires, automatically promote related traces/logs (based on traceID, error signature) to hot tier for N days.- Pre-warmed playbooks: store mappings from alert signatures → relevant query templates; automated runbooks retrieve compressed context.- Links & breadcrumbs: ensure logs include traceIDs and spanIDs so any log search can jump into trace UI.

Forensics & long-term ability- Keep aggregates (per-service p50/p95/p99, error rates by signature) daily for entire retention.- Store canonical error signatures and examples to reproduce or debug later.- Provide rehydration API with SLA (e.g., <24h for specific traceID; <72h for bulk).

Trade-offs & rationale- Cost vs fidelity: Full retention forever is prohibitively expensive. Hot tier preserves fidelity when it matters (recent incidents). Warm/cold trade lower cost for higher latency and reduced fidelity.- Complexity vs control: Hybrid sampling and auto-promotion add complexity but enable smart cost control and maintain debuggability for failures.- Latency vs availability: Archiving to Glacier reduces cost but increases restore latency; retain indices/metadata cheaply to avoid blind searches.- Risk: Sampling risks missing rare bugs — mitigated by tail-based retention (keep all errors/p99 traces) and retention of metadata to rehydrate targeted traces.

Example numbers (illustrative)- Hot: 30 days full = ~40% cost- Warm: 11 months sampled/errors = ~40% cost- Cold: multi-year aggregates + archives = ~20% costAdjust sampling rates and retention windows per service criticality and SLOs.

Implementation checklist- Ensure all logs include traceID/spanID and structured fields.- Implement tail-based sampling in tracing pipeline.- Build auto-promotion on alert rules.- Provide rehydration workflows + quotas and monitoring for archival retrieval costs.- Review quarterly, tune sampling by service/error signature.

This approach balances cost and debuggability: immediate full context for recent incidents, targeted retention of problematic traces, and long-term aggregate data and rehydration paths for forensic needs.

HardSystem Design

33 practiced

Design a distributed rate-limiting system that supports per-tenant dynamic quotas and can handle 100k requests/s globally with low latency. Discuss trade offs between centralized Redis-based counters, token buckets at the edge, and approximate algorithms like leaky-bucket sketches.

Sample Answer

Requirements & constraints:- Functional: per-tenant dynamic quotas (can change in seconds), enforce hard + soft limits, burst support per tenant.- Non-functional: 100k req/s global, P99 latency <5ms for check path, high availability, eventual consistency acceptable for soft limits, strong for hard limits.

High-level architecture:- Edge nodes (INGRESS) near clients run a local token-bucket for low-latency checks and short bursts.- Central Quota Store: Redis cluster (sharded, HA) storing authoritative tenant quotas and long-term counters.- Global Aggregator (streaming): Kafka + stateless processors to aggregate edge usage and reconcile with Redis periodically.- Control Plane: APIs to update tenant quotas; changes pushed to edges via config stream (gRPC/websocket).- Metrics & Alerting: Prometheus/Grafana, SLOs, burn-rate alerts.

Approach trade-offs:

1) Centralized Redis counters- How: every request increments a per-tenant counter with TTL or Lua script for atomic check-and-incr.- Pros: Strong correctness, easy to change quotas centrally.- Cons: High RPS load on Redis (100k/s with many tenants), network latency, single region bottleneck. Requires heavy sharding, client-side batching, pipelining, and connection pooling. Need fallback when Redis unavailable (deny-safe or degrade to soft limits).- Best when correctness for hard limits is critical and write load manageable.

2) Token buckets at the edge- How: Each edge keeps a token-bucket per tenant (refilled by rate and synced periodically with central quota). Local decision for immediate accept/reject.- Pros: Lowest latency, scales linearly with edges, reduces central load, handles bursts well.- Cons: Requires careful reconciliation to prevent overallocation. Quota changes need fast propagation; risk of temporary overuse. Need periodic lease/check-in to central store to correct drift.- Best when low latency and capacity to accept minor eventual consistency are required.

3) Approximate algorithms (leaky-bucket sketches / probabilistic counters)- How: Use approximate counters (Count-Min Sketch, reservoir sampling) at edges and aggregator for global estimation.- Pros: Extremely scalable, low memory, low central traffic, useful for massive tenant sets and cold tenants.- Cons: False positives/negatives; harder to enforce strict hard limits. Complexity in error bounds and compensating logic.- Best for analytics, throttling cold tenants, or providing soft throttles.

Hybrid recommended design (SRE-balanced):- Primary enforcement: Edge token-buckets for fast path. Tokens computed from authoritative quotas and replenishment rate; allow small burst cushion.- Secondary authoritative enforcement: Redis-backed strict counters invoked when edges detect near-limit (e.g., >90%) or for critical tenants. Use Lua scripts for atomic decrement/check.- Reconciliation: Edges emit usage to Kafka every ~1s. Aggregators update Redis and detect drift; control plane pushes quota updates and corrective tokens.- Graceful degradation: If Redis unavailable, edges operate in conservative mode (reduce allowed tokens) and emit alerts.- Operational concerns: autoscale Redis shards, enforce connection limits, use region-local Redis replicas with consistent hashing; add chaos tests, SLA-based tenant prioritization, rate-limiter metrics per-tenant, and automated remediation runbooks.

Key SRE practices:- Define SLOs (e.g., false-enforcement rate <0.1%, latency targets).- Canary quota updates and tiered rollout.- Heavy observability: per-tenant burn-rate, edge drift, reconciliation lag, Redis latency/inflight ops.- Capacity planning: simulate 2x expected RPS, account for skew (hot tenants), shard hot-tenant keys separately or use token-server for them.

Conclusion: For 100k req/s with low latency and dynamic quotas, an edge-first token-bucket design with Redis as authoritative backstop and streaming reconciliation offers the best balance of latency, correctness, and scalability. Use approximate techniques only where strict correctness is not required.

EasyTechnical

28 practiced

Explain the difference between Service Level Objectives (SLOs), Service Level Agreements (SLAs), and error budgets from an SRE perspective for an online payments platform. Describe how you would set a numeric SLO for payment success rate, calculate a one-month error budget, and how the team should act when the budget is being consumed.

Unlock Full Question Bank

Get access to hundreds of Architecture and Technical Trade Offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.