Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.
MediumTechnical
125 practiced
How do you design consistent cross-system backups for an application composed of multiple storage systems: relational DB for profiles, object storage for files, and a cache layer for sessions? Describe a strategy to ensure a consistent point-in-time restore across systems and how CDC or transaction IDs can help.
Sample Answer
Start by clarifying the goal: a point-in-time consistent restore across 3 systems (relational profiles, object storage files, session cache). The core idea is to coordinate snapshot/backup epochs using a common timeline marker (transaction ID, LSN, or monotonic timestamp) and use CDC to record or re-order changes that cross system boundaries.Strategy:- Choose a canonical source of truth for ordering (DB WAL/LSN or a global logical clock). For relational DB, capture a consistent snapshot and note its LSN/commit position.- For object storage, when writing/overwriting files include the DB transaction ID or a write-timestamp in metadata. Take object-store snapshots or copy-on-write with the same epoch marker.- For cache (sessions), either persist sessions to a durable store periodically, or treat cache as ephemeral and rebuild from DB/events. If durable, flush/serialize sessions and tag with the same epoch.Using CDC/transaction IDs:- Enable CDC on DB to stream changes with LSNs. CDC events can carry references to object writes (file IDs) created during the same transaction.- When initiating a coordinated backup, record the DB LSN = X. Then ensure object snapshots include only objects with metadata <= X (or copy the universe as of X). CDC can later be used to replay events between object snapshots and DB to reconcile small timing gaps.Implementation pattern:1. Start backup: record DB LSN X (use read-consistent snapshot).2. Pause or quiesce application writes briefly OR use transactional markers so new writes include LSN > X.3. Snapshot DB at LSN X.4. Snapshot object storage as of X (object metadata stamped with LSN or timestamp). If object store lacks snapshot, use manifests with CDC cross-references.5. Persist cache state labeled X, or rebuild on restore.Restore:- Restore DB snapshot at LSN X.- Restore objects present at X (or replay object CDC up to X).- Restore/reconstruct sessions consistent with X (or accept loss if ephemeral).- If you have CDC logs, replay events up to X to reconstruct any out-of-band references.Trade-offs and best practices:- Short quiesce windows reduce complexity but require app support for stamping writes with transaction IDs.- If you cannot atomically coordinate, rely on CDC and idempotent replay to converge to a consistent state.- Test restores regularly; store manifests and checksums; encrypt and automate the process.This approach ensures a repeatable point-in-time restore by binding all writes to a common ordering (LSN/txid) and using snapshots + CDC replay when full atomic coordination is impractical.
HardTechnical
73 practiced
You operate a distributed key-value store using leader election and synchronous replication. A network partition causes a minority partition to believe it has leadership and serve writes, causing divergence. Describe how to detect this situation, mitigate ongoing damage, and restore the cluster to a consistent state while minimizing data loss.
Sample Answer
Situation: A partitioned minority believes it’s leader and accepts writes, causing split-brain divergence in a synchronously-replicated KV store.Detection:- Monitor quorum and term/epoch metadata: detect nodes accepting writes when cluster-wide quorum not held (e.g., write accepted with term T but majority of nodes report a different leader/term).- Compare committed indices/log hashes across replicas (periodic checksum/gossip). Divergence shows mismatched commitIndex or log hashes.- Alert on inverted quorum: nodes in minority still serving client writes (client errors, increased inconsistency metrics).Mitigation (stop ongoing damage):1. Immediately fence off suspected rogue leaders: - Increment global epoch/lease on majority side and refuse any write requests signed by older epochs. - Require clients to present current epoch token; minority will be rejected once token enforced.2. Stop accepting new writes on minority partition: - Automatic: once minority can’t reach quorum, its RPCs to leader election should fail; proactively shut write API or put node into read-only mode. - Operational: push configuration to set nodes into read-only or maintenance mode if automatic fencing unavailable.3. Rate-limit or redirect clients: - Return errors/redirects instead of silently accepting conflicting writes. - Use client libraries that check leader epoch and retry against known majority leader.Restoration / Reconciliation:1. Snapshot and collect logs: - Take immutable snapshots of diverged nodes and collect their WALs/operation logs and vector clocks.2. Choose reconciliation authority: - Prefer majority-committed state as canonical if using a strict quorum/majority-commit model (this minimizes data loss). - If minority writes are high-value, run a conflict-resolution process instead of blind discard.3. Reconcile operations: - If system uses CRDTs or commutative ops: merge automatically (apply CRDT merge). - Otherwise, use operation logs + causal metadata (term, commitIndex, client timestamp, version vectors): - For operations that commute or are idempotent, reapply missing ops to majority replicas. - For conflicting keys, surface conflict resolution policies: - Application-driven merge (best). - Rule-based (last-writer-wins by hybrid logical clock), or manual review for critical keys.4. Re-integrate nodes: - Replay compatible minority ops onto majority leader in controlled replay window, with strict validation and tombstone handling. - Once reconciliation completes and logs converge, update commitIndex and let followers catch up via normal replication.5. Validate and promote: - Run consistency checks (hashes, sampling) to confirm convergence. - Remove read-only/fenced state and restore normal leader election.Minimizing Data Loss — practical trade-offs:- Design-time choices reduce loss: prefer majority-committed durability, use leases/fencing, attach causal metadata (vector clocks / HLC), and prefer mergeable data models (CRDTs).- If majority-wins: document and notify users about potential loss of minority writes; provide tools to recover per-key where possible.- If preserving minority writes: accept operational delay and human-in-the-loop resolution to avoid data corruption.Key implementation patterns:- Enforce leader epochs/fencing tokens at RPC/auth layer.- Maintain per-key version vectors or HLC timestamps to support deterministic merges.- Periodic Merkle-tree checksums per shard to detect divergence quickly.- Client libraries that validate leader epoch and automatically retry.This approach detects split-brain early, stops further divergence via fencing and read-only modes, then reconciles using logs and causal metadata—preferring automatic merges where safe and human review where necessary—minimizing data loss by design and operational process.
EasyTechnical
67 practiced
What are health checks in production systems? Describe liveness, readiness, and deep application health checks, including examples of endpoints, metrics, and the difference in action taken by orchestration systems upon failing each type.
Sample Answer
Health checks are automated tests that tell orchestration and monitoring systems whether an app is healthy enough to receive traffic or needs intervention. They help maintain availability and enable automated recovery.Liveness check:- Purpose: Detects if the process is alive or stuck (deadlock, crash-loop).- Example endpoint: GET /healthz or /live that returns 200 if process responsive.- Metrics to consider: process uptime, event-loop responsiveness, heartbeat.- Orchestrator action: If liveness fails (non-200 / timeout), Kubernetes restarts the container (livenessProbe → restart).Readiness check:- Purpose: Indicates whether the instance is ready to serve requests (warm-up, dependency unavailable).- Example endpoint: GET /ready or /readyz that returns 200 only when app initialized and dependencies connected.- Metrics: DB connection pool healthy, cache warmed, config loaded, feature flags applied.- Orchestrator action: If readiness fails, the pod is removed from service endpoints / load balancer but not restarted; traffic is drained until readiness returns.Deep (application) health checks:- Purpose: Richer diagnostics beyond simple up/down — verify downstream services, data integrity, performance thresholds.- Examples: POST /deep-health that runs queries against DB, calls critical downstream APIs, checks message queue depth.- Metrics: query latency, error rates, queue backlog, percent of failing transactions.- Action: Typically used by monitoring/alerting (Prometheus + Alertmanager) to trigger alerts, paging, or automated mitigation (circuit-breaker, scaled replicas). Orchestrator may use them as readiness signals if configured, but because they’re heavier they’re usually sampled or run on a schedule.Best practices:- Keep liveness cheap and fast; readiness can be slightly heavier but still quick.- Avoid deep checks on every request; run periodically or asynchronously.- Return clear JSON with status+details for observability.- Tie metrics to SLIs/SLOs and use alert thresholds rather than raw failure counts.
HardTechnical
89 practiced
Design an observability plan specifically to validate disaster recovery correctness. What automated checks, synthetic tests, and metrics would you run before, during, and after a failover to declare success? Include both technical and business-level validations and how you would automate them in CI/CD pipelines.
Sample Answer
Clarify requirements:- RTO/RPO targets, critical business transactions, dependencies (databases, queues, caches), failover scope (region, AZ, cluster), and allowed blast radius. Define success SLAs (technical + business).Pre-failover (readiness checks, automated in CI/CD pipeline stage “DR-validate”):- Infrastructure: IaC plan-diff + automated drift detection (Terraform plan + Sentinel/OPA policies).- Replica health: replication lag, WAL/CDC lag < threshold.- Backups: latest backup age & integrity check (test-restore metadata).- Config & secrets: secrets availability in target vault.- Synthetic smoke: deploy test instance in target region, run end-to-end synthetic transaction (auth -> create -> read -> delete) against read-replicas.- Business metric sanity: synthetic revenue/order throughput baseline run to capture expected QPS/latency.During failover (automated orchestration + monitoring):- Control-plane checks: DNS TTL swap completion, LB target health, routing table updates — verify via automated probes.- Latency & error-rate probes: distributed synthetic clients hitting primary endpoints and critical APIs every 30s; alert if 95th pct latency > SLA or error rate spike > x%.- Data correctness probes: shadow-write verification—write to both primary and DR target if possible and compare checksums.- Queue/backpressure checks: consumer lag, queue depth trends.- Business KPI probes: synthetic checkout completion, cart abandonment rate, sign-in conversion.Post-failover (validation & declare success):- Functional: run full smoke + regression suite (automated tests in pipeline), validate critical flows return expected results.- Data integrity: reconcile record counts for critical tables, checksum comparison (sampled + full for small datasets).- Performance: load test at expected production QPS for 15–30min; measure error rate, latency P50/P95/P99.- Business-level: verify synthetic revenue matches expected window within tolerance; order-to-fulfillment pipeline observable (no stuck orders).- Consistency: check no message duplication or loss (idempotency counters, dedupe logs).Automation approach:- Encode checks as pipeline jobs (e.g., Jenkins/GitHub Actions/GitLab CI) triggered by scheduled DR rehearsals and pre-cutover gates. Use feature flags to run destructive steps only in rehearsal windows.- Use orchestration tool (Argo Workflows / Step Functions) to run steps with automatic rollback on failure.- Use monitoring/alerting platforms (Prometheus + Alertmanager, Datadog) to collect metrics and evaluate SLOs via recorded queries; failures block CI job.- Persist DR run artifacts: test results, snapshots, diffs; generate automated DR runbook with timestamped evidence for audits.Pass criteria (example):- All pre-failover readiness checks green.- No data loss detected on sampled reconciliation.- Error rate <= baseline + 2x for transient 5min window, then back to baseline.- Business synthetic success rate >= 99% during and after failover.If any gate fails, automatic rollback and postmortem job that collects logs, traces (distributed tracing), and root-cause artifacts.This plan ensures technical correctness and business continuity by combining deterministic checks, active synthetic validation, automated orchestration in CI/CD, and clear pass/fail gates with rollback.
HardTechnical
69 practiced
A distributed cache (e.g., Redis cluster) holds ephemeral but critical data needed to restore service behavior quickly. During failover to a secondary region the primary cache is lost. Describe options to make caches recoverable, including write-through and write-behind patterns, async replication, and fallback strategies. Discuss performance and cost trade-offs and how to choose an approach.
Sample Answer
Situation: When a primary-region Redis cluster is lost during failover, clients need access to critical ephemeral data so service behavior remains correct or can be quickly restored. There are multiple recoverability patterns — each trades latency, durability, complexity and cost.Options- Write-through: every cache write synchronously writes to backing store (DB). Pros: strong durability and easy recovery (replay from DB). Cons: higher write latency and DB load; simpler to implement.- Write-behind (async write): cache acknowledges writes immediately and batches/flushed to DB asynchronously. Pros: low latency and reduced DB calls. Cons: potential data loss on cache failure, complexity in ordering/flush guarantees; requires durable queue or UPSERT semantics to avoid loss.- Async cache replication across regions: cross-region replication (CRR) of cache entries (e.g., Redis replication or CDC to replicate writes to a secondary cluster). Pros: fast local reads after failover; lower DB reliance. Cons: eventual consistency, higher network egress cost, increased complexity for conflict resolution.- Persistent snapshots/AOF: enable RDB/AOF to persist cache state to disk and ship snapshots to secondary. Pros: simpler recovery; cons: slower, snapshot windows can lose recent writes, storage/transfer cost.- Hybrid: write-through for critical keys, write-behind for noncritical; replicate hot keys only.Fallback strategies- Read-through with DB fallback: on cache miss, read from DB; combine with stale-while-revalidate to serve slightly stale data while refreshing.- Cache warming and bootstrapping: pre-populate critical keys in secondary region during deployment or predicted failover windows.- Feature flags / degraded mode: reduce feature surface that depends on cache during failover.- Durable write queue: write operations append to a durable queue (Kafka) that can rebuild cache in target region.Trade-offs and decision factors- RPO/RTO requirements: if RPO=0, prefer synchronous persistence or cross-region replication; if short RTO acceptable, async + rebuild may suffice.- Latency sensitivity: low-latency writes favor write-behind + durable queue; read-latency critical systems may require cross-region replication.- Cost: cross-region replication and increased DB load cost more; snapshots and async pipelines add storage/transfer costs.- Complexity & operational burden: synchronous patterns simpler to reason about; distributed replication needs monitoring, conflict resolution.- Consistency model: choose eventual vs strong consistency depending on correctness requirements.Recommendation approach1. Classify keys by criticality.2. For critical keys: write-through or synchronous replication + multi-AZ/region replication.3. For less-critical: write-behind with durable queue and periodic snapshots; employ read-through fallback and cache warming.4. Validate with failure drills, measure RTO/RPO, tune TTLs and replication windows, and monitor costs to iterate.
Unlock Full Question Bank
Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.
