Disaster Recovery and Business Continuity Questions

Designing and maintaining plans, architectures, and processes to ensure service continuity and recoverability after major incidents or disasters. Topics include defining Recovery Time Objective and Recovery Point Objective, conducting business impact analysis and tiering services by criticality, dependency mapping and recovery ordering, selecting replication and backup strategies including synchronous and asynchronous replication, active active and active passive topologies, snapshots and transaction log based point in time recovery, and planning cold, warm, and hot recovery sites. Also covers failover and failback procedures, orchestration and automation of recovery workflows, runbook creation and stakeholder roles and communications, regular disaster recovery testing and exercises including tabletop, simulated failover, full recovery drills and chaos engineering, metrics tracking such as mean time to recovery and actual Recovery Time Objective achieved, off site and geographic redundancy considerations, cloud versus on premise trade offs, regulatory and data residency requirements, and postexercise reviews to close recovery gaps.

MediumTechnical

27 practiced

Your company is considering using cloud as a disaster recovery site for an on-prem primary. Compare cloud vs on-prem approaches for cold, warm, and hot recovery models. Address costs, provisioning speed, data transfer, security, and testing trade-offs, and recommend an approach for a mid-size startup.

Sample Answer

Clarify requirements: RTO/RPO targets, data volume, compliance, budget, and recovery test frequency. I'll compare cloud vs on‑prem across cold, warm, hot DR models and conclude with a recommendation for a mid‑size startup.

Cold (restore from backup):- Cloud: Low ongoing cost (object storage), slower provisioning (minutes→hours to restore VMs/DBs), large data egress costs on recovery; easy to automate backups and geographic diversification; security relies on IAM, encryption-at-rest/in-transit. Testing: snapshot/restore drills are straightforward but slower.- On‑prem: Low latency for restore if spare hardware exists; higher capital cost to maintain idle hardware; provisioning speed depends on available spare capacity; testing needs scheduled use of secondary hardware.

Warm (standby services, partial replication):- Cloud: Moderate cost (reserved or autoscaling smaller instances + cross‑region replication), fast provisioning (scale up instances, DNS failover), data transfer steady (replication costs + sync bandwidth), strong managed replication options (RDS, Aurora replicas); security benefits from provider tooling but requires secure networking (VPC, peering). Testing: failover drills easier with isolated environments.- On‑prem: Medium cost to maintain hot spares or clustered sites; provisioning slower than cloud if manual; lower egress costs; more ownership of replication stack and security appliances.

Hot (active-active, synchronous replication):- Cloud: High cost (duplicate full capacity), near‑zero RTO and minimal RPO with multi‑region architectures, provisioning immediate; data transfer costs high due to continuous replication; managed services simplify consistency; security and compliance easier with provider features but you must design for multi‑AZ/R region networking. Testing: continuous failover testing possible with less impact.- On‑prem: Extremely expensive (secondary data center), complex networking and WAN provisioning, same ownership of security; testing is complex and disruptive.

Trade-offs summary:- Costs: cloud lowers capital expense and enables pay‑as‑you‑go; on‑prem has higher upfront CAPEX but predictable OPEX.- Provisioning speed: cloud > on‑prem (automation, APIs).- Data transfer: cloud incurs egress and cross‑region costs; on‑prem may have predictable bandwidth charges.- Security: both viable; cloud provides managed controls but requires strong IAM/network design and vendor trust; on‑prem gives full control but higher maintenance.- Testing: cloud simplifies automated, repeatable drills; on‑prem tests are more manual, riskier.

Recommendation for a mid‑size startup:Adopt a cloud‑based tiered approach: cold backups in multi‑region object storage for archival; warm standby for critical services (use managed DB replicas, smaller instance families with IaC to scale on failover); reserve hot/active‑active only for highest‑SLA components where business needs justify cost. This balances cost, speed, and reliability while enabling frequent, low‑risk testing and rapid provisioning via automation (Terraform, CI/CD). Include encryption, strict IAM, network segregation, and regular DR runbooks and audits.

EasyTechnical

28 practiced

Explain Recovery Time Objective (RTO) and Recovery Point Objective (RPO). Give concrete examples: choose RTO/RPO targets for a customer-facing payments API versus a nightly analytics/reporting job, and explain trade-offs in cost, complexity, and user impact when tightening RTO/RPO.

EasyTechnical

23 practiced

Explain snapshot-based backups versus transaction-log (WAL) based point-in-time recovery for relational databases. Describe a step-by-step process to restore a PostgreSQL instance to a specific timestamp using base backups plus WAL, and list common pitfalls (e.g., LSN mismatches, inconsistent snapshots).

Sample Answer

Snapshot-based backups vs WAL-based PITR:

- Snapshot backups capture the full data directory state at a point in time (e.g., filesystem snapshot or cold copy). Fast to create/restore as a whole, but only recoverable to the snapshot timestamp. Consistency depends on how the snapshot was taken (cold vs hot with fsync/quiesce).- Transaction-log (WAL) based Point-In-Time Recovery (PITR) combines a base snapshot (base backup) plus archived WAL segments to replay changes up to an arbitrary timestamp/LSN. This enables fine-grained recovery (e.g., just before a user error) and continuous recovery between base backups.

Step-by-step: restore PostgreSQL to a specific timestamp using base backup + WAL

1. Ensure you have: - A consistent base backup (made with pg_basebackup or via filesystem snapshot while PostgreSQL was quiesced) - Complete WAL archive covering from base backup’s start LSN through desired target time

2. Prepare data directory: - Stop PostgreSQL on the recovery host. - Remove/rename existing data directory and extract the base backup into PGDATA (ensure correct ownership/permissions).

3. Configure recovery (Postgres 12+): - Create a file postgresql.conf edits as needed. - Create a file recovery.conf (pre-12) or place a recovery.signal and use postgresql.conf in 12+; set: - restore_command = 'cp /wal_archive/%f %p' (or a script to fetch WAL files) - recovery_target_time = '2025-05-01 14:23:00' (or recovery_target_lsn) - recovery_target_action = 'promote' (or 'pause') Example (postgresql.conf or recovery.conf):

bash

restore_command = 'bash /usr/local/bin/restore-wal.sh %f %p'
   recovery_target_time = '2025-05-01 14:23:00'
   recovery_target_action = 'promote'

4. Start PostgreSQL: - Start the server. It will restore WAL files via restore_command and replay up to the target. Monitor logs for progress.

5. Promote when recovery completes (if not auto-promoted): - Use pg_ctl promote or create trigger file (in older configs). Verify consistency and run integrity checks.

6. Post-recovery: - Reset archive_mode/restore settings as needed. - Run validation (checksums, application smoke tests) and reconfigure clients.

Common pitfalls and how to avoid them

- Inconsistent base backup (hot snapshot without checkpoint/fsync): ensure base backup is consistent — use pg_basebackup or filesystem snapshots coordinated with pg_start_backup/pg_stop_backup or use database-aware snapshot tooling.- Missing WAL segments: recovery halts if WAL needed for target time is absent. Verify WAL retention/archival and test restores regularly.- LSN/timeline mismatches: base backup has a start LSN and timeline; if you restore a base from an older timeline but fetch WALs from a diverged timeline, recovery fails. Keep timeline/history (.history files) and use correct archived WALs or pg_wal restore scripts that handle timelines.- Incorrect restore_command permissions/paths: restore script must be executable and able to write to PGDATA. Test restore_command manually.- Recovery target too early/late: recovery_target_time must be within WAL coverage. Use recovery_target_lsn if precise.- Leftover files from previous cluster (postmaster.pid, old configs): start from a clean PGDATA.- Confusing pre-12 vs post-12 recovery flow: newer Postgres uses recovery.signal/standby.signal and postgresql.auto.conf; follow version-specific docs.

Best practice: automate base+WAL archival, enforce retention/monitoring of WAL archival, run regular recovery drills, and store base backups and WAL archives in redundant offsite storage.

HardSystem Design

31 practiced

Design a system that continuously verifies backups and restores by performing automated periodic restores into an isolated environment, validating data integrity, schema compatibility, and performance. Discuss scaling, cost, data masking, and how to handle failures detected by verification.

Sample Answer

Requirements:- Functional: periodic automatic restores from backups into isolated environments; validate data integrity (checksums, row counts), schema compatibility (migrations, constraints), and performance (query latencies, throughput).- Non-functional: safe (masked PII), scalable to many backups and DB types, low cost, observable, actionable alerts, auditable.

High-level architecture:1. Orchestrator (Kubernetes CronJobs / Airflow) schedules verification pipelines.2. Immutable Backup Fetcher pulls backup artifacts from object storage (S3/GCS).3. Isolated Restore Environment: ephemeral infra per run—namespaced Kubernetes cluster or separate VPC with DB instances spun via IaC (Terraform/CloudFormation) or using cheap managed replicas/snapshots.4. Restore Engine executes DB restore (native restore tools), runs migrations if needed.5. Validation Suite: - Integrity: verify checksums, row counts, FK/PK presence, sample record hashes. - Schema: run schema-diff tools, run migrations in dry-run mode, validate constraints. - Performance: synthetic workload runner (rally/jmeter/custom) measuring key queries/transactions against SLOs.6. Data Masking Layer: apply deterministic or non-reversible masking/synthetic data transformation after restore but before exposure to validators. Use tokenization or format-preserving masking; keep masking as IaC step.7. Results Collector: stores outcomes, metrics, logs in time-series DB (Prometheus/Grafana), and artifacts in object storage.8. Alerting & Incident Flow: failures create PagerDuty incidents, auto-created Jira for triage; optionally block backups from promotion pipeline.

Scaling:- Parallelize verification pipelines across namespaces/workers; use queueing (RabbitMQ/SQS) to feed restore jobs. Use horizontal autoscaling for workers.- For large databases, perform selective/partial restores for quick checks (metadata + sampled partitions) and full restore on lower cadence.- Prioritize critical datasets with higher verification frequency; use backoff policies.

Cost control:- Use spot/ephemeral instances; tear down infra immediately after runs.- Tiered verification: cheap sanity checks every hour, full restore nightly or weekly.- Use incremental or copy-on-write snapshot restores (RDS snapshots, ZFS clones) to shorten time and cost.- Retention: store only verification artifacts and metrics, not full restored DBs.

Data masking & compliance:- Mask as close to the restore process as possible, enforce via automated policy engine. Maintain audit logs proving masking applied.- For tests requiring real data patterns, use synthetic data generators to preserve distributions without PII.

Failure handling:- Classify failures: transient (infra/timeouts), data issues (corruption), schema drift, performance regressions.- Transient: automatic retry with exponential backoff and alternative restore node.- Data corruption: mark backup as suspect, escalate to SRE/backup owner, quarantine backup set, trigger re-run from earlier snapshot.- Schema drift: notify owners, run compatibility report, optionally auto-apply migration if safe flag set.- Performance regressions: compare against baseline, open performance incident, attach workload trace and query plan to ticket.- Maintain incident playbooks, runbooks and automated rollback gating for backups used in production restores.

Observability & KPIs:- SLOs: verification success rate, time-to-detect, mean-time-to-remediate.- Metrics: run duration, resource cost per run, failure reasons, query latency deltas.- Dashboards and weekly reports; maintain audit trail for compliance.

Trade-offs:- Full restores are costly/time-consuming vs. sampled checks faster but less thorough.- Masking complexity vs. developer access—use role-based access and dedicated sanitized test clusters.

This design provides automated, scalable verification while minimizing cost and preserving privacy; failures are classified, automated where safe, and routed to human operators with rich diagnostics for rapid remediation.

MediumTechnical

22 practiced

Design a tabletop exercise simulating a regional outage that takes down authentication and payments. Define participants, objectives, injects, success criteria, communication plan, timeboxed steps, and what artifacts you will collect to measure readiness.

Sample Answer

Scenario summary: A regional cloud provider outage causes loss of auth (OAuth/token service) and payment gateway reachability for Region A; traffic fails over to Region B but tokens are invalid and payments timeout, impacting checkout and login for ~40% of users.

Participants:- Incident Commander (SRE lead)- Auth service owner (backend engineer)- Payments owner (payments engineer)- Traffic/Load Balancing owner (SRE)- Deploy/Runbook SME (SRE)- Customer Support lead- Product/PM representative- Security on-call- Communications/PR

Objectives (SMART):- Verify failover procedures restore login and checkout in Region B within 60 minutes in the exercise- Validate detection/alerting triggers for simultaneous auth+payments failures- Test cross-team communications and runbook clarity- Identify at least 3 actionable gaps in automation or runbooks

Injects (timeboxed):0m — Monitoring alert: spike in 5xx from auth and PAYMENT_TIMEOUT on checkout (start)10m — Customers report new account logins failing; error rate increases20m — Region A declared degraded by provider; managed failover begins but tokens rejected in Region B35m — Payment processor API returns 503 and queued transactions appear50m — Security flags unusual token reissuance requests (simulate potential abuse)70m — Legal/PR asks for statement draft

Success criteria:- Teams declared a coordinated mitigation and executed failover steps per runbook within 60 minutes- Auth tokens validated or alternative session flow implemented such that >90% of checkout attempts succeed in Region B within 90 minutes- Clear communications sent to stakeholders within SLA windows (see plan)- Post-exercise: ≥3 prioritized fixes logged and owners assigned

Communication plan:- Use dedicated incident channel (Slack) and locked Zoom for coordination- IC posts status every 15 minutes; a run cadence of 15m during critical, then 30m- Customer Support uses prepared templated messages; PR drafts external statement within 90m- Escalation matrix and decision authority documented beforehand

Timeboxed steps (90-minute exercise):- 0–10m: Triage and role assignment, confirm facts- 10–25m: Execute detection & initial mitigations (traffic reroute, circuit breakers)- 25–50m: Auth and payments owners run playbooks (token refresh strategies, payment queueing/fallback)- 50–75m: Implement workarounds (e.g., temporary guest checkout, manual settlement)- 75–90m: Stabilize, communicate status, capture decisions

Artifacts to collect:- Timeline of alerts, actions, and decisions (incident timeline)- Runbook execution logs and checklist completion status- Monitoring/metrics snapshots (error rates, latency, token issuance rates, payment success)- Communication artifacts (messages, PR drafts, support templates)- Post-mortem draft with identified gaps, root causes, action items, and owners

Measurement of readiness:- Time-to-detect, time-to-ack, time-to-mitigate (against targets)- Percentage of runbook steps automated vs manual- Number and severity of uncovered gaps- Confidence survey from participants (qualitative readiness)

Run follow-up: 1-week action review, implement top 3 fixes, re-run a focused micro-exercise for auth or payments within 8 weeks.

Unlock Full Question Bank

Get access to hundreds of Disaster Recovery and Business Continuity interview questions and detailed answers.

Join thousands of developers preparing for their dream job.