Disaster Recovery and Business Continuity Questions

Designing and maintaining plans, architectures, and processes to ensure service continuity and recoverability after major incidents or disasters. Topics include defining Recovery Time Objective and Recovery Point Objective, conducting business impact analysis and tiering services by criticality, dependency mapping and recovery ordering, selecting replication and backup strategies including synchronous and asynchronous replication, active active and active passive topologies, snapshots and transaction log based point in time recovery, and planning cold, warm, and hot recovery sites. Also covers failover and failback procedures, orchestration and automation of recovery workflows, runbook creation and stakeholder roles and communications, regular disaster recovery testing and exercises including tabletop, simulated failover, full recovery drills and chaos engineering, metrics tracking such as mean time to recovery and actual Recovery Time Objective achieved, off site and geographic redundancy considerations, cloud versus on premise trade offs, regulatory and data residency requirements, and postexercise reviews to close recovery gaps.

EasyTechnical

23 practiced

Describe how you would conduct a basic Business Impact Analysis (BIA) for a product line. Include which stakeholders you'd interview, key questions to assess criticality, how you'd map technical dependencies, and how to document recommended RTO/RPO tiers based on business impact.

Sample Answer

**Approach overview**I’d run a concise BIA in 3 phases: stakeholder interviews, technical dependency mapping, and produce recommended RTO/RPO tiers with justification.

**Stakeholders to interview**- Product Manager (business priorities, SLAs)- Engineering Lead / Tech Lead (features, acceptable downtime)- DevOps / SRE (current recovery capabilities)- Support/CS Ops (customer impact, incident volume)- Sales/Finance (revenue & contractual obligations)- Security/Compliance (regulatory requirements)

**Key questions to assess criticality**- What revenue, legal, or customer-impact occurs if this product is down for 1h, 8h, 24h?- Which users/customers are affected (all vs subset)?- Which features must be available to consider the product “operational”?- Are there contractual SLA penalties or regulatory limits?- What is acceptable data loss? (seconds/minutes/hours)- What manual workarounds exist and their cost/time?

**Mapping technical dependencies**- Inventory components: web frontend, API services, DBs, caches, message queues, third‑party APIs, CI/CD pipeline, auth.- Create dependency graph (service -> datastore -> network -> external).- Annotate each node with owner, current HA, backup cadence, and estimated recovery steps (runbook).- Identify single points of failure (e.g., single-region DB, single NFS).

**Documenting RTO/RPO tiers**- Define 3 tiers with examples and rationale: - Tier 1 (Critical): RTO ≤ 1h, RPO ≤ 1m — customer-facing payments, auth, order processing. Requires multi-AZ/region DB replication, automated failover, tested runbooks. - Tier 2 (Important): RTO ≤ 8h, RPO ≤ 1h — core APIs, analytics ingestion. Requires replicas, backups hourly, scripted restore. - Tier 3 (Non-critical): RTO ≤ 24–72h, RPO ≤ 24h — reporting, batch jobs. Daily backups, manual restore.- Include cost/benefit rationale and implementation actions (backup frequency, cross-region replication, CI/CD rollback strategies, chaos testing cadence).- Deliverables: BIA summary, dependency map diagram, prioritized RTO/RPO table, implementation roadmap and runbooks for Tier 1 services.

MediumTechnical

25 practiced

Design a dashboard and metric set to measure actual RTO and recovery progress during a disaster recovery drill. Identify data sources (monitoring events, health checks, DNS or load-balancer changes), KPIs to show per-service recovery progress, visualizations for blockers, and how the dashboard supports go/no-go decisions by incident commanders.

Sample Answer

**High-level approach**Design a single “DR Drill Operations” dashboard that maps target RTOs to live recovery signals, surfaces blockers, and produces a clear go/no-go recommendation for the incident commander (IC).

**Data sources**- Health checks / readiness probes (K8s liveness/readiness, app HTTP 2xx)- Monitoring events & alerts (Prometheus, Datadog, CloudWatch)- DNS / Load‑balancer changes and propagation (Route53 / ALB logs, TTLs)- Storage/replication status (DB replica lag, snapshot completion)- Orchestration jobs (ArgoCD/Jenkins pipelines, recovery playbook steps)- Network checks (ICMP/endpoint synthetic tests, service mesh)- Incident runbook progress (ticketing status, manual checkboxes)

**KPIs / per-service metrics**- Actual RTO elapsed vs target RTO (min)- Percentage recovered: healthy_instances / expected_instances- Time to first successful transaction (end‑to‑end smoke)- Replication lag (seconds)- Deployment/rollout success rate (%)- Config change propagated (yes/no + TTL remaining)- Blocker count and severity

**Visualizations**- Top row: global status tiles (Overall RTO % complete, elapsed RTO, recommended go/no‑go)- Per-service rows: progress bars showing % recovered, mini-timeline of events, green/yellow/red status- Timeline chart: events plotted (failover start, DNS change, health check pass)- Heatmap: services by recovery progress to highlight slowest- Blocker panel: active blockers with owner, ETA, impact, logs/links- Alert stream + latest synthetic test results- Confidence gauge combining automated checks + manual confirmations

**How it supports go/no‑go**- Explicit thresholds: go if Overall % recovered >= X, all P0 services green, and no P0 blockers open- “Must-pass” checks: list of binary checks (DNS propagation, DB primary writable, smoke test) that must be true- IC view: one-click evidence links (logs, probe traces, runbook items) and a confidence score (weighted by KPIs)- Controls for staged decisions: allow partial go (read‑only traffic, limited LB weights) with rollback triggers and automated runbook steps

**Example decision rule**- Go if: all P0 services 100% recovered, replication lag < 5s, smoke test success, blocker count = 0. Otherwise hold and escalate with blocker remediation steps shown.

HardTechnical

30 practiced

Design a chaos engineering experiment to validate the RTO for database failover in a production-like environment. Define the hypothesis, planned experiments, safe blast-radius limits, observability to collect, rollback procedures, and quantitative success/failure criteria for the experiment.

Sample Answer

**Hypothesis**If we trigger an automated database failover in a production-like environment using the same control plane and traffic patterns as production, the system will recover within the defined RTO (target: 2 minutes) without violating SLOs (error rate <1%, p95 latency degrade <200ms) and without data loss (no committed transactions lost).

**Planned experiments (progressive)**1. Dry-run (non-disruptive): run failover automation against a standby replica in staging to validate scripts, IAM, and runbooks.2. Canary (5% traffic): in production-like cluster, divert 5% of read/write traffic to an isolated tenant or canary namespace; trigger failover of the primary for that shard.3. Progressive ramp (25%, 100% within maintenance window): if canary passes, increase to larger subset.4. Full-shard test during low-traffic window: trigger primary termination and observe automatic failover.

Tools: Terraform/Ansible for infra, Kubernetes jobs or AWS SSM to stop primary, Route53 weighted records / Envoy for traffic split, chaos tool (Litmus/Chaos Mesh), CI to run playbooks.

**Safe blast-radius limits**- Limit to single shard/region or single tenant- Start at 5% traffic and max 25% for non-blocking period- Maintenance window, on-call engineer present, kill-switch to abort automation- Read replicas kept cold backups; backups snapshot before experiment

**Observability to collect**- Time-series: primary down timestamp, failover initiation, promotion completion- RTO metric: time from failure to first successful write on new primary- DB metrics: replication lag, WAL apply rate, commit acknowledgements- App metrics: request rate, error rate (5xx), p95/p99 latency- Infrastructure: CPU, I/O, connection counts, DNS propagation times- Logs/traces: distributed traces showing error propagation and retries- Alerts/incident timelines (automated tickets)

**Rollback / abort procedures**- Immediate traffic cut: switch weighted DNS to healthy replicas or route traffic to standby region- Promote original primary if safe (follow runbook: restore WAL/replica sync)- Restore from snapshot only if corruption detected- Post-mortem checklist: revert schema changes, rebuild replicas, re-run consistency checks- Automate kill-switch via CI pipeline to halt chaos controller and revert routing

**Quantitative success/failure criteria**Success:- RTO measured ≤ 2 minutes (time between primary failure and first successful committed write on new primary)- Application error rate increase ≤ 1% above baseline during failover- p95 latency increase ≤ 200 ms- Replication lag at failover ≤ 5 s; no missing commits detected by consistency checks- No customer-facing data loss

Failure:- RTO > 2 minutes- Error rate spike > 1% or sustained >5 minutes- p95 latency degrade > 200 ms or p99 spikes causing user impact- Detected data loss or corruption- Manual intervention required longer than runbook thresholds

Run experiment iteratively, collect telemetry, refine automation and runbooks, and re-run until repeatable success.

EasyTechnical

37 practiced

Which monitoring metrics and alert types would you configure to detect a disaster-level degradation for a multi-tier web application? Cover application-layer (error rates, latency), infrastructure-layer (host/instance health, disk, IOPS), data-layer (replication lag, queue depth), and synthetic checks. Give example thresholds and how alerts should be routed to incident response.

EasyTechnical

23 practiced

Define Mean Time To Recovery (MTTR) in the context of disaster recovery and business continuity. Explain how to measure MTTR during DR drills and live incidents using monitoring timestamps and runbook checkpoints, how to attribute downtime to specific services, and how MTTR differs from the RTO agreed in business requirements.

Unlock Full Question Bank

Get access to hundreds of Disaster Recovery and Business Continuity interview questions and detailed answers.

Join thousands of developers preparing for their dream job.