Cost and Complexity Trade Offs in High Availability Questions
Understanding that extreme reliability comes at cost (redundancy, replication, multiple regions). At Staff level, discuss how to make informed tradeoffs: not everything needs to be multi-region active-active. Discuss tiered approaches where different services have different SLOs based on business criticality and cost.
HardSystem Design
36 practiced
Compare client-side replication (clients write to nearest region and sync later) versus single-master-per-service with regional caches for a low-latency global application. For each approach discuss consistency semantics, conflict resolution complexity, operational cost, and how to quantify when one wins over the other.
Sample Answer
Requirements and constraints (implicit): global low-latency writes/reads, high availability, SLOs for read/write latency and consistency, expected write volume and conflict rate, fault domains (region outages, partitioning).High-level summary:- Client-side replication: clients write to local region (edge), changes async-synced to other regions (multi-master-ish, eventual consistency).- Single-master-per-service + regional caches: one region owns writes for a shard/service; other regions serve reads from cache/replica and proxy or forward writes to master.Comparison by axis:1) Consistency semantics- Client-side replication: eventual consistency by default. Reads at local region may not reflect remote writes until sync. Causality can be preserved if you implement vector clocks or CRDTs, but strong consistency is not provided without coordination.- Single-master: strong consistency for writes within a shard (linearizable or at least read-after-write if routed), assuming single master synchronously applies updates. Reads from cache may be stale unless cache invalidation or read-through to master is used.2) Conflict resolution complexity- Client-side replication: high complexity. Concurrent divergent writes across regions require conflict detection + resolution. Choices: - Last-writer-wins (LWW): simple but may lose data. - Application-level merge rules or CRDTs: correct but often complex to design and test; CRDTs increase payload/metadata and have semantics constraints. - Human/manual reconciliation: costly operationally.- Single-master: low complexity. Master serializes updates; conflicts are naturally avoided. If master unavailable, writes must fail or be buffered at client/proxy (adds complexity).3) Operational cost & runbook complexity- Client-side replication: - Higher operational cost: data reconciliation tools, extra telemetry to detect divergence, more complex backups/rollbacks, larger testing surface (network partitions). - Monitoring needs: per-region replication lag, conflict rates, CRDT metadata growth, convergence SLA. - More tolerant to region failures for local availability but recovery/merge ops can be heavy.- Single-master + caches: - Lower conflict ops but higher networking cost (cross-region latency for writes) or complexity in routing writes to master. - Need cache invalidation, TTL tuning, and cache warm-up strategies. - Simpler runbooks for correctness; failover complexity: master failover/leader election, data re-sharding, and ensuring cache coherence after failover.4) Latency, availability trade-offs- Client-side replication wins for ultra-low local write latency and availability during partitions (AP in CAP). Good when app can tolerate eventual consistency and conflicts are rare or domain supports CRDTs (e.g., counters, presence).- Single-master wins when correctness is prioritized (CP for strong consistency) and write latencies to master are acceptable or can be mitigated (write coalescing, regional proxies, smart partitioning).5) How to quantify when one wins (metrics & thresholds)- Define SLOs: p95/p99 read latency, p95/p99 local write latency, data-staleness SLA (e.g., 99% of reads reflect latest write within X seconds), acceptable conflict-rate (% of writes that require manual reconciliation), availability target.- Measure: - Local write latency distribution under both designs. - Replication convergence time distribution (client-replication). - Conflict incidence rate and operational cost per conflict (hours to resolve, business cost). - Cross-region write latency and its variance (single-master). - Cost: infra (extra storage/metadata), engineering time (CRDT design/ops), network egress.- Decision rule examples: - If business requires <50ms local write p95 and conflict tolerance <0.01% and CRDTs not feasible → single-master likely unsuitable unless you add local buffering; client-replication favored. - If staleness must be <100ms globally or conflicts would cause >$X loss per occurrence → single-master with regional proxies or multi-region active-passive with synchronous or semi-sync replication. - Compare total cost = infra + expected operational MTTR * incident cost + latency-penalty (revenue impact). Whichever has lower expected annual cost and meets SLOs wins.Operational recommendations (SRE lens)- If choosing client-replication: invest in strong observability (replication lag, divergence probes), automated reconciliation pipelines, bounded metadata (compact vector clocks/CRDTs), clear runbooks for merge/rollback, simulate partitions in chaos testing.- If choosing single-master: optimize routing (anycast/proxies), regional write coalescing, fast leader election, cache invalidation patterns (push invalidation or short TTLs), circuit breakers to prevent cascading failures when master is overloaded.- Hybrid: use client-replication for low-risk, commutative data (counters, presence) and single-master for critical, high-conflict data (financial transactions).Conclusion: pick client-replication when ultra-low local latency and partition tolerance matter more than strict global consistency and conflict overhead is manageable (or CRDTs apply). Pick single-master when correctness and low conflict complexity matter and cross-region write latency is acceptable or can be mitigated. Quantify with SLO-driven metrics and cost-of-conflict calculations to make the decision evidence-based.
HardSystem Design
51 practiced
Compare regional sticky sessions (routing a user to their home region) with a global session store for a high-traffic global web application. Analyze tradeoffs in terms of availability, consistency, cost, complexity, and user experience. Propose a hybrid approach that balances these concerns and justify your recommendation.
Sample Answer
Requirements and constraints:- Global read/write traffic at high volume, low perceived latency (<100ms target), 99.95% availability, session state (auth + small user prefs) must be available on request, tolerance for eventual consistency on non-critical fields, budget constraints.High-level comparison1) Regional sticky sessions (route a user to home region)- Availability: High if region healthy; resilient to cross-region outages because sessions local. But catastrophic region failure means user sessions lost unless replicated.- Consistency: Strong within region (single-writer); trivial consistency model.- Cost: Low storage/replication cost (no global replication); lower cross-region network egress.- Complexity: Simple routing + health checks; fewer distributed storage concerns.- UX: Low latency for home users; poor if user travels or region unavailable (forced re-login).2) Global session store (multi-region replicated DB / globally-consistent KV)- Availability: Can be highly available if replicated; but dependent on global interconnect—network partitions can degrade availability or require read-only modes.- Consistency: Can provide strong consistency (CP) or tunable (AP) depending on design (Spanner vs Dynamo-style). Strong consistency increases latency.- Cost: Higher (cross-region replication, egress, more complex DB).- Complexity: High operational complexity: consensus, replication lag, conflict resolution.- UX: Good for geo-mobility; consistent experience across regions, but added latency for writes if using strong consistency.Trade-offs summary:- Sticky favors availability, low cost, low complexity, but weak geo-mobility and region-failure recovery.- Global store favors consistent multi-region access and mobility, but costs more, adds latency and operational risk.Hybrid approach (recommended for SRE balance)- Primary design: Regional session caches + asynchronous cross-region replication + optional global fallback. - Each region writes sessions to a local durable store (fast, single-writer). - Asynchronously replicate session deltas to an eventually-consistent global store (object store or multi-region KV) for cross-region failover and mobility. - Use sticky routing by default to keep latency low. - If a user's home region is unhealthy or user originates from a different region for > N minutes, read from global store (stale-read tolerant, with read-repair). - For critical fields (auth tokens, revocations), use a small strongly-consistent control plane: replicate writes to a geo-consensus store with faster quorum (e.g., regional quorum + leader) to ensure security-sensitive data is consistent. - Short TTLs and versioning to limit conflict window; merge-on-read policy and last-writer-wins with timestamps for non-critical prefs.Justification (SRE focus)- Balances latency and availability: sticky routing keeps common-path latency low; global async replication enables resilience to region failures and supports mobility.- Controls cost: avoids full synchronous global replication; only critical bits use strong consistency.- Operational complexity: manageable—local stores are simple; global component limited in scope (fallback and critical control plane). Monitoring surfaces: replication lag, region health, cache hit rates, conflict rates. SLOs: set SLIs for session availability and stale-read probability; alert on replication lag or failover events.- Failure modes & mitigations: on region outage, failover reads to global store with circuit-breaker; on replication lag, serve local stale sessions but enforce short TTLs and re-auth where necessary; run regular chaos tests.Implementation notes- Use local Redis/managed cache + durable backing (regional SQL or NoSQL).- Replication: stream changes via CDC or pub/sub to global storage (S3 + index or multi-region Dynamo).- Control-plane: small strongly-consistent store (Spanner/etcd with regional placement) for tokens/revocations.- Instrumentation: latency percentiles, replication lag, failover frequency, user impact metrics.This hybrid delivers low-latency UX and high availability while bounding cost and limiting global-consistency complexity; it provides clear operational levers for SREs to tune behavior under load and failure.
EasyTechnical
39 practiced
Compare active-active and active-passive multi-region deployment strategies. For each describe the operational complexity, costs (compute, networking, data replication), and the failure modes they mitigate. Give two concrete scenarios where you would prefer each strategy.
Sample Answer
Active-active vs active-passive — quick SRE comparison.Active-active- Operational complexity: High. Requires global traffic routing, conflict-free data replication (leaderless or multi-leader with conflict resolution), consistent health checks, and runbooks for partial-region failures and failback.- Costs: Higher compute (full capacity in multiple regions), higher networking (cross-region load balancing and egress), continuous data replication (multi-master or CDC) and monitoring costs.- Failure modes mitigated: Region failover with near-zero downtime, better read/write locality, reduced latency for global users. Does not eliminate consistency anomalies (split-brain, write conflicts).- Prefer when: 1) Global user base with strict latency SLOs (e.g., social platform with real-time feeds). 2) Revenue-critical system needing near-continuous availability (e.g., payment gateway across regions).Active-passive- Operational complexity: Lower day-to-day complexity but needs reliable failover automation, tested runbooks, and regular DR drills to keep passive replicas warm.- Costs: Lower compute (passive scaled down), lower networking and replication often via async backups or log shipping (cheaper), but potential warm-standby costs if faster RTO required.- Failure modes mitigated: Protects against full-region outage and data loss if replication is synchronous; reduces blast radius. Higher RTO and potential data loss with async replication.- Prefer when: 1) Cost-sensitive service with moderate availability requirements (e.g., batch analytics service). 2) Internal tools or admin dashboards where occasional minutes-hours downtime is acceptable but DR protection is needed.Trade-offs summary: active-active buys lower RTO/latency at higher operational and financial cost and complexity; active-passive reduces cost and complexity but increases RTO and potential consistency risk. Choose based on SLOs, budget, and operational maturity.
MediumTechnical
40 practiced
Which SLIs would you select for a public user-facing HTTP API versus a background batch processing service to inform cost and complexity tradeoffs? For each SLI explain measurement method, sampling frequency, and how it should influence redundancy decisions.
Sample Answer
For a public user-facing HTTP API:- Latency (p95/p99 of successful requests): measure at the ingress/load‑balancer or API gateway using request end-to-end timings (time-to-first-byte and full response). Sample every request if feasible; otherwise 1:10 sampling with deterministic hashing to avoid bias. Influence on redundancy: high tail latency suggests insufficient capacity or noisy neighbors — justify horizontal redundancy (more instances behind LB), autoscaling targets, and regional failover to reduce p99.- Availability / Success rate (HTTP 2xx ratio vs total requests): measured at ingress by status codes and client-perceived errors (include network/timeout). Continuous per-minute rolling windows. If availability drops near SLO, add redundancy (multi-AZ), health checks, circuit breakers; low-impact errors might be resolved by retries rather than added infra.- Error budget burn (rate of user‑impacting errors): derived from SLO vs observed errors; evaluated hourly. Drives risk tolerance for deployments and whether to invest in active-active redundancy vs cheaper single-region with aggressive rollback.For a background batch processing service:- Throughput / Job completion rate (jobs/sec or jobs per window): measured at job scheduler level (worker count vs queue depth). Sample per job event, aggregate per minute. If throughput shortfall is frequent, prefer vertical scaling or worker pools before global multi-region complexity.- Job success rate / Data correctness (percentage of successful completed jobs): measured by job outcomes and end-to-end business validations. Daily aggregation often sufficient unless business impact is real-time. Low success rates push for retry logic and idempotency; only if retries fail repeatedly invest in redundant processing lanes or separate hot path.- Latency for SLA-bound batches (time-to-complete median / 95th): measured from enqueue to final commit; sample all jobs for critical pipelines, otherwise 1:100 for non-critical. If tail latency is high and deadlines matter, add parallelism or partitioned processing; for non‑critical jobs accept longer windows to save costs.General guidance:- Match SLI fidelity to user impact: user-facing traffic → per-request high-frequency metrics; background jobs → aggregated, windowed metrics.- Use deterministic sampling to preserve statistical soundness.- Let SLOs and error-budget burn rate decide redundancy investment: active redundancy for high user impact; retries, backoff, and simpler scaling for lower-impact batch workloads.
MediumSystem Design
54 practiced
Design a global API gateway approach that balances cost and availability. Compare three options: a single central gateway, multiple regional gateways with global DNS failover, and client-side region selection. For each option evaluate latency, cost, complexity, and operational concerns, and recommend one for a consumer-facing application with global traffic.
Sample Answer
Requirements (clarify): global consumer-facing app needs low latency worldwide, high availability (multi-region failover), cost-conscious, operational simplicity for SREs, and consistent security/policy enforcement (auth, rate-limit, WAF). Target SLO: 99.95% global and <150ms p95 latency.Option A — Single central gateway- Latency: High for distant users; single hop to central region adds RTTs → poor p95 outside region.- Cost: Lower infra overhead (one cluster), cheaper TLS cert management.- Complexity: Low deployment complexity; simple routing and policy management.- Operational concerns: Single region outage or network partitions impact all users; requires strong CDN fronting to cache static responses; harder to meet availability SLO without multi-region global infra.- When suitable: small scale, non-latency-sensitive apps.Option B — Multiple regional gateways + global DNS failover (e.g., geo-DNS + health checks)- Latency: Low — users route to nearest gateway; best p95.- Cost: Higher — run gateway clusters per region, replicate configs and certs; cross-region control plane costs.- Complexity: Medium-high — need config sync (GitOps), global traffic management (Route53/GSLB), consistent policies, can use IaC and central control plane.- Operational concerns: Rolling upgrades coordination, regional failover via DNS has TTL propagation delays; need health checks and fast failover (short TTLs, Anycast + BGP or GSLB recommended). Observability must aggregate across regions.- When suitable: balanced availability/latency for global user base.Option C — Client-side region selection (client picks nearest endpoint via CDN/latency check)- Latency: Potentially best if client correctly measures; but variability and stale client logic can cause suboptimal routing.- Cost: Lower server infra (fewer regions), more complexity in client releases and retries.- Complexity: High app-side logic, version skew handling, security/token scope challenges, harder to change routing centrally.- Operational concerns: Harder to debug (client variability), increases risk of misrouting post-deploy; reliance on client platform updates.Recommendation for consumer-facing global app:Choose Option B (multiple regional gateways + global traffic management) as primary. It provides low latency and high availability, matches SLOs, and keeps control plane centralized. Mitigations: use Anycast or GSLB with health checks and short TTLs, automate config sync via GitOps and CI/CD, front gateways with CDN for static cache and TLS offload, implement global observability (traces, aggregates), and run cross-region chaos tests. Use Option C sparingly for advanced mobile optimizations (optional fallbacks), and avoid a single central gateway except for prototypes.
Unlock Full Question Bank
Get access to hundreds of Cost and Complexity Trade Offs in High Availability interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.