Reliability, High Availability, and Tradeoffs Questions

Design patterns and decision making for ensuring availability correctness and graceful behavior under failure while balancing technical trade offs. Topics include redundancy and failover strategies active passive and active active deployments; fault isolation using bulkheads and circuit breaker patterns; graceful degradation and feature gating strategies; defining and mapping service level objectives and service level agreements to recovery point and recovery time objectives; multi region and multi availability zone deployment considerations; testing for reliability including chaos engineering and fault injection; and reasoning about consistency versus availability trade offs and the operational cost of stronger guarantees. Candidates should be able to choose reliability patterns to meet business objectives and to explain their implications for cost performance and maintainability.

EasyTechnical

27 practiced

You're rolling out a new enrichment stage that improves event records but increases processing latency. Explain strategies for graceful degradation and feature gating so the system preserves availability if the enrichment stage becomes slow or fails. Describe toggle granularity, fallback behavior, observable signals to monitor, and automatic rollback policies.

Sample Answer

Situation: We need an enrichment stage that improves event records but adds latency. The goal is to preserve pipeline availability if it slows or fails.

Toggle granularity:- Global kill-switch (immediate system-wide disable).- Service/cluster-level toggles (disable per region or processing cluster).- Stream/topic or pipeline-stage toggles (only for specific event types).- Per-tenant or per-customer toggles for graduated rollouts and limited impact.- Canary flags for a small % of traffic (1–5%) for validation.

Fallback behavior:- Synchronous fallback: if enrichment times out, attach a lightweight "unenriched" marker and pass the original event downstream.- Default-stub enrichment: return cached or best-effort values (last-known-good) to avoid nulls.- Async enrichment: persist event and enqueue for background enrichment; downstream consumers get baseline data immediately and enriched copy later.- Backpressure controls: degrade optional enrichment gracefully before blocking upstream (e.g., drop enrichment for lower-priority events).

Observable signals to monitor:- Latency percentiles (p50/p95/p99) of enrichment calls and end-to-end processing time.- Error rate and timeouts of enrichment service.- Queue/backlog length and consumer lag (Kafka partitions, topic lag).- Throughput and dropped events.- Downstream SLA violations and business metrics (e.g., ETL freshness, failing downstream jobs).- Resource signals: CPU/memory, thread pool saturation, connection pool exhaustion.

Automatic rollback / mitigation policies:- Circuit breaker: open circuit when error rate or p99 latency exceeds thresholds for X minutes; automatically half-open after cooldown and re-evaluate on canary traffic.- Auto-toggle: feature flag system triggers global or scoped disable when combination of signals (error rate > 2%, p99 > SLA, backlog growth > threshold) persists for N minutes.- Gradual ramp-down: reduce enrichment percentage progressively (100% → 50% → 10% → 0%) while monitoring impact.- K8s liveness/health probes and automated redeploys if service unhealthy; but prefer feature-flag rollback rather than redeploy on data correctness incidents.- Alerting + runbook: automated paging when auto-rollback occurs with clear next steps; record events for postmortem.

Why this approach:- Granular toggles let us limit blast radius and test safely.- Fallbacks preserve availability and dataflow correctness.- Observability ensures quick detection and targeted response.- Automated circuit breakers and progressive rollbacks balance reliability with safety while preserving business continuity.

MediumSystem Design

23 practiced

Design a highly available ingestion architecture able to accept 100k events per second and meet 99.95% availability. Specify component choices (API gateway, load balancing, queueing system, stream buffer, consumer autoscaling), data guarantees (at-least-once vs best-effort), how to handle bursts and backpressure, and how you would test availability targets.

Sample Answer

Requirements & constraints:- Ingest 100k events/s sustained, 99.95% availability (~22 min downtime/month), multi-AZ, support bursts, prefer durable ingestion for downstream processing.

High-level architecture:Client -> API Gateway (edge) -> Regional Load Balancer -> Ingress layer (stateless HTTP workers) -> Durable streaming buffer (Kafka / AWS MSK / Confluent Cloud or Kinesis) -> Consumer fleet (stream processors) -> Downstream sinks (S3, data lake, analytics)

Component choices & rationale:- API Gateway: AWS API Gateway / GCP Cloud Endpoints for authentication, rate limiting, WAF at edge.- Load Balancer: NLB/ALB in front of ingress autoscaling group to distribute TCP/HTTP concurrently across AZs.- Ingress workers: lightweight autoscaling HTTP services (K8s Deployment/HPA or ECS) that validate, enrich minimal metadata and produce to streaming system.- Queue/Stream buffer: Apache Kafka (MSK) or Kinesis. Kafka recommended for high throughput, partitioning, configurable replication, and consumer offset control.- Storage sink: S3 as cold durable landing for batch/DR; Kafka Connect to S3.- Consumer autoscaling: Consumers in K8s with HPA based on consumer lag (Kafka lag) and CPU; use partition-aware consumers (consumer groups = #partitions). Scale partitions to meet throughput.

Data guarantees:- At-least-once by default: ingress writes synchronously to Kafka with retries; consumers commit offsets after processing. Document possibility of duplicates; provide idempotency keys downstream for dedupe.- For low-latency non-critical events, offer a best-effort endpoint that returns quickly and uses async batching to reduce backpressure.

Handling bursts & backpressure:- Overprovision and partition capacity: provision sufficient partitions (e.g., 1000+) so parallelism supports peak throughput.- Token-bucket rate limiting in API GW and per-client quotas to protect backend.- Ingress workers batch produce to Kafka (compression) to increase throughput.- Configure Kafka with adequate brokers, replication factor (>=2-3), and retention to absorb bursts.- Overflow cold path: when streaming cluster is saturated, fallback to durable object store (S3) as write-ahead buffer via signed PUTs or Kinesis Firehose; separate job ingests those files into Kafka.- Backpressure propagation: ingress returns 429 when queues exceed safe thresholds; use exponential backoff clients and circuit-breaker patterns.

Availability & resiliency:- Multi-AZ deployment for API GW, LB, brokers and storage. Kafka with replication.factor=3, min.insync.replicas=2.- Use cross-region clusters or MirrorMaker for region failover if required.- Monitoring/alerts: latency, publish error rate, consumer lag, broker health, ISR count, partition under-replicated, GC pauses.- Automated failover: broker replacement automation, rolling upgrades, rack-awareness.

Testing to meet 99.95%:- Load testing: run gradual soak tests up to 120k-200k eps with realistic payloads; verify latency percentiles (p50/p95/p99).- Chaos & fault injection: simulate AZ failures, broker failures, network partitions, API GW throttling, consumer crashes (Chaos Engineering).- Game days: rehearse failover procedures and recovery time objectives.- SLO/SLA measurement: instrument end-to-end availability and request success rate; collect metrics for 30/90-day windows to prove 99.95%.- Failure mode tests: full Kafka cluster restart, sustained burst beyond buffer capacity to verify fallback to S3 path and consumer catch-up.

Operational considerations:- Capacity planning: map event size -> throughput bytes; size partitions and brokers accordingly.- Cost/ops tradeoffs: more partitions/brokers = higher cost but easier to meet availability.- Idempotency and dedupe libraries for downstream consumers.- Security: TLS, auth, client quotas, encryption-at-rest.

This design balances durability (at-least-once), high throughput via partitioning and batching, burst absorption via S3 fallback, and availability via multi-AZ replication and chaos-tested operational runbooks to hit 99.95%.

EasyTechnical

20 practiced

Consider this simplified architecture: Ingest API -> Load Balancer -> 2 replicas of ingest-service -> Kafka cluster (single Zookeeper and single broker) -> Consumer group -> Data warehouse. Identify single points of failure, explain the impact of each, and propose practical fixes (no code required) to remove or mitigate those SPOFs while weighing operational cost.

Sample Answer

Situation: Given the pipeline Ingest API -> LB -> 2 ingest-service replicas -> Kafka (1 ZK, 1 broker) -> consumer group -> data warehouse.

Single points of failure, impact, and practical mitigations:

1) Load balancer (single)- Impact: LB outage prevents all client traffic reaching services.- Fix: Use a managed HA load balancer (cloud provider ALB/NLB) or run LB in active-active across AZs with health checks. Cost: low–medium (managed LB cost), high reliability for little ops overhead.

2) Ingest-service placement (only 2 replicas, possibly same host/AZ)- Impact: If both pods/VMs or their AZ fail, ingestion stops; if services are stateful, failover is harder.- Fix: Ensure replicas scheduled across multiple AZs/nodes, add autoscaling (HPA) and at least 3 replicas. Keep services stateless (or externalize state). Cost: low (extra instances), improves availability.

3) Kafka broker (single) and Zookeeper (single)- Impact: Broker or ZK failure halts ingestion/persistence; data loss risk, no leader failover.- Fix: Move to an HA Kafka cluster: minimum 3 brokers and 3 ZK nodes (or use Kafka Raft (KRaft) to avoid ZK). Set topic replication factor >=2-3 and configure min.insync.replicas to avoid data loss. Option: use managed Kafka (AWS MSK, Confluent Cloud) to reduce ops. Cost: medium–high (more nodes, storage), but essential for durability.

4) Consumer group (single consumer instance or single point in processing)- Impact: If consumer process fails, downstream processing halts; if consumers commit offsets unsafely you can lose/duplicate data.- Fix: Run consumers in a scalable consumer group with multiple instances across AZs, implement idempotent writes, checkpointing, and poison-message handling. Cost: low–medium.

5) Data warehouse (single node or single region)- Impact: Warehouse outage prevents writes/queries, blocking ingestion or downstream analytics; risk of losing recent data.- Fix: Use HA/replicated DW (multinode cluster, read replicas, cross-region replication) or buffer failed writes (retry queue / S3 as landing bucket) to avoid blocking ingestion. Cost: medium–high depending on vendor.

6) Network/region-level outage- Impact: Regional failure affects entire pipeline.- Fix: Design for multi-region failover for critical data (replicate Kafka clusters or use cross-cluster mirroring, replicate DW). Use durable buffering (object storage) for cross-region sync. Cost: high—only for critical SLAs.

Operational recommendations:- Monitoring, alerting, and runbooks for component failures.- Backpressure and retry strategies so downstream outages don't drop data (use durable buffer like S3 or a separate persistent queue).- Prefer managed services if team ops bandwidth is limited—higher cost but lower operational risk.

Trade-offs: Basic mitigations (multi-AZ, extra replicas, managed LB) get most availability gains at modest cost. Full HA for Kafka and multi-region DR are costlier but necessary if data loss/unavailability has high business impact.

HardTechnical

27 practiced

As a senior data engineer leading a small reliability team with limited budget, propose a prioritization framework and KPIs to decide which reliability engineering work to pursue across multiple services. Include how to estimate ROI, factor in business impact, and communicate trade-offs and timelines to stakeholders.

Sample Answer

Framework (prioritization lens)1. Risk × Impact × Effort matrix: - Risk: likelihood of failure or data loss (scale 1–5). - Business Impact: downstream cost if failure occurs (revenue, SLAs, analyst hours) (1–5). - Effort: engineering cost in person-weeks (1–5). - Priority score = (Risk × Business Impact) / Effort. Tiebreaker: strategic alignment.

2. Time-to-value gating: - Quick wins (≤2 wks): automation, alert tuning, flaky-job fixes. - Mid-term (2–8 wks): idempotency, retry logic, schema evolution handling. - Long-term (>8 wks): platform redesign, multi-region replication.

KPIs to track- Mean Time to Detect (MTTD) for pipeline failures- Mean Time to Recover (MTTR) for data availability- Data Availability SLA (% of successful on-time runs)- Number of incidents per month and incident severity-weighted score- Downstream impact: analysts’ blocked hours saved, business revenue at risk- ROI-related: cost saved per incident, cost per prevented outage

Estimating ROI- Quantify incident cost: avg. revenue impact or business-hours lost per outage + engineering incident response cost.- Estimate reduction from project: e.g., alerting + auto-retry reduces incidents by X% (use historical data).- ROI = (annual incident cost avoided) / (project cost in $ or engineer-weeks).- Present best-case / expected / worst-case scenarios for assumptions.

Factoring business impact- Map services to stakeholders and revenue/analytics dependency.- Use dependency graph to score downstream users and SLA criticality.- Add regulatory/compliance multiplier where applicable.

Communicating trade-offs & timelines- Use a one-page roadmap per quarter: prioritized backlog with score, estimated effort, expected reduction in MTT R/incident cost, and confidence level.- Present options: “Low-cost high-impact” vs “High-cost strategic” with timelines and risk of not doing work.- Run monthly triage with stakeholders to re-score when incidents or business priorities change.- Use pilot/proof-of-value for larger efforts with measurable success criteria (reduce MTTR by X%, cut blocked analyst-hours by Y).

Example (concise)- Problem: recurring late-run causing 8 hrs of analyst downtime monthly (~$12k/month).- Fix: add idempotent retries + alerting (2 weeks, $8k).- Expected reduction: 80% fewer downtime hours → $9.6k/month saved → payback <1 month; priority: high.

This approach makes prioritization data-driven, ties engineering work to business outcomes, and keeps stakeholders aligned with transparent trade-offs and measurable success.

MediumTechnical

20 practiced

A nightly batch job aggregates data and must complete within a 2-hour SLA. Compare deploying it across multiple availability zones in a single region versus a multi-region deployment to improve availability. Discuss failover approaches, data locality implications, cost, deployment complexity, and recovery time trade-offs.

Sample Answer

Situation: You run a nightly ETL/aggregation job that must finish within a 2-hour SLA. You’re choosing between deploying across multiple AZs in one region or across multiple regions to improve availability.

Short answer: For most batch jobs with strict SLA but large data locality needs, a multi-AZ single-region deployment is simpler, lower-cost, and faster to recover from typical infrastructure failures. Multi-region improves resilience against region-wide outages but increases complexity, cost, data replication latency, and recovery time trade-offs.

Comparison (key dimensions):

- Failover approaches - Multi-AZ (single region): Use active-active or active-standby across AZs. Orchestrator (Airflow/Kubernetes) retries tasks in other AZs automatically; shared regional storage (EBS, EFS, S3) remains available. Failover is fast (minutes) for instance failures; orchestrator state replication is straightforward. - Multi-region: Options are active-passive (primary region runs job, secondary stands by with replicated data/catalog) or active-active with partitioned processing. Failover includes DNS/coordination changes and promoting secondary pipelines—longer, more complex (tens of minutes to hours).

- Data locality implications - Multi-AZ: Low latency to regional data stores; no cross-region transfer. Ideal if source data and downstream consumers are regional. - Multi-region: Requires cross-region replication of raw data, metadata, and state (e.g., S3 CRR, database replication). Replication lag can mean secondary region has stale data—risk to SLA and correctness for time-sensitive aggregation.

- Cost - Multi-AZ: Lower cost—no cross-region egress, single-region storage, simpler monitoring. - Multi-region: Higher cost—cross-region replication, duplicate compute or warm-standby resources, additional networking and operational overhead.

- Deployment complexity - Multi-AZ: Simple CI/CD, single config, unified IAM and monitoring. Easier testing and rollback. - Multi-region: Complex deployment pipelines, cross-region secrets, multi-region orchestration, compliance differences; higher testing burden.

- Recovery time trade-offs - Multi-AZ: Fast recovery for node/AZ failures; if entire region fails, recovery requires manual or automated failover to another region and will likely exceed the 2-hour SLA unless pre-provisioned warm-standby exists. - Multi-region: Better protection against region-wide outages but only meets 2-hour SLA if you maintain warm-standby or fully active pipelines in both regions—higher cost. Cold-starting in another region (provisioning infra + warming caches) can exceed SLA.

Recommendation (Data-engineer view):- If primary risk is AZ or instance failure and data gravity is high, choose multi-AZ single-region with robust retries, job checkpointing, idempotent processing, and monitoring/alerts. Add automated instance replacement, horizontally scalable compute (Spark on EMR/EKS), and small warm-standby if business requires.- If region-wide outages are frequent/catastrophic for business continuity, invest in multi-region with continuous replication and either warm-standby pipelines or partitioned active-active processing. Prioritize deterministic replayability, global metadata replication, and test failovers regularly.

Operational best practices:- Make pipelines idempotent and checkpoint progress to resume mid-run.- Measure replication lag and test end-to-end failover drills.- Use cost/ROI analysis: estimate incremental cost of warm-standby vs cost of SLA breach.- Automate health checks and have runbooks for failover to reduce human recovery time.

Unlock Full Question Bank

Get access to hundreds of Reliability, High Availability, and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.