Disaster Recovery and Business Continuity Questions

Designing and maintaining plans, architectures, and processes to ensure service continuity and recoverability after major incidents or disasters. Topics include defining Recovery Time Objective and Recovery Point Objective, conducting business impact analysis and tiering services by criticality, dependency mapping and recovery ordering, selecting replication and backup strategies including synchronous and asynchronous replication, active active and active passive topologies, snapshots and transaction log based point in time recovery, and planning cold, warm, and hot recovery sites. Also covers failover and failback procedures, orchestration and automation of recovery workflows, runbook creation and stakeholder roles and communications, regular disaster recovery testing and exercises including tabletop, simulated failover, full recovery drills and chaos engineering, metrics tracking such as mean time to recovery and actual Recovery Time Objective achieved, off site and geographic redundancy considerations, cloud versus on premise trade offs, regulatory and data residency requirements, and postexercise reviews to close recovery gaps.

EasyTechnical

29 practiced

Explain synchronous versus asynchronous replication for storage or databases. Discuss how each approach affects RPO, write latency, throughput, and availability. Provide examples of scenarios where synchronous replication is impractical and suggest mitigations or hybrid approaches.

MediumTechnical

26 practiced

Design a secure key management and encryption approach for backups replicated across regions. Address KMS key policies and separation of duties, key rotation, cross-region key availability and failover during DR, secure key export policies or escrow, and procedures to avoid scenarios where keys are unavailable during a provider outage.

Sample Answer

**Approach summary**Design layered KMS + HSM-backed keys with explicit policies, separation of duties, automated rotation, multi‑region availability, and an escrow/DR policy so backups remain decryptable during provider outages.

**KMS key policies & separation of duties**- Use customer-managed keys (CMKs) in KMS backed by Cloud HSM for high assurance.- Enforce least privilege via IAM + key policy: distinct roles for Key Admin (create/rotate/grant), Crypto Operator (rotate/monitor), and Backup Service (use for encryption/decryption). No single role can both export/escrow and authorize restores.- Require MFA and Just‑In‑Time (JIT) approvals for sensitive key grants.

**Key rotation**- Automate rotation on schedule (e.g., 90 days) with staged rewrap: generate new data encryption keys (DEKs), re-encrypt metadata and rotate CMK aliases to minimize outage.- Maintain old keys for retention period to decrypt older backups; automate expiry and safe purge.

**Cross-region availability & DR failover**- Use multi‑region (replicated) CMKs where provider supports (e.g., AWS multi‑region keys) or maintain pairwise-synchronized CMKs in each region.- Replicate only wrapped DEKs, not plaintext keys. During failover promote target-region CMK and update backup metadata to point to local CMK.- Health-check automation: if primary region KMS unreachable, automatic retry window, then failover to replicated CMK and notify security ops.

**Secure key export / escrow**- Prohibit unrestricted key export. For escrow, use split‑knowledge export stored in external HSMs or an offline Hardware Security Module under dual control (e.g., 2-of-3 quorum).- Escrow operations require multi-party approvals, tamper-evident logging, and periodic escrow drills.

**Avoiding key-unavailability during provider outage**- Maintain replicated CMKs in at least two independent regions/providers where feasible (multi-cloud HSM escrow).- Implement automatic retry/backoff and local cached wrapped DEKs to permit restores for a defined window if KMS temporarily unreachable.- Practice DR runbooks: periodic restore drills, smoke tests, and documented rollback to escrowed keys if provider SLA fails.

**Audit, monitoring, and governance**- Enforce immutable audit logs (CloudTrail + SIEM), real‑time alerts on key policy changes, and quarterly audits. Document key lifecycle, escrow SOPs, and run regular DR exercises.

This design balances security (HSM, split knowledge, no plaintext export) with availability (multi-region keys, cached wrapped DEKs, escrow) and clear separation of duties so backups remain recoverable during outages.

HardTechnical

25 practiced

For a high-write transactional database, analyze trade-offs between synchronous cross-region replication and local asynchronous replication. Provide a model for estimating added commit latency from network RTT, impact on throughput, expected bandwidth cost, and propose mitigations such as batching, WAN acceleration, and hybrid semi-synchronous approaches.

Sample Answer

**Clarify scope & assumptions**- High-write OLTP workload (small writes ~1–4KB), strong consistency goal across two regions, baseline local DB commit latency Llocal, network RTT = RTT, available bandwidth B, replication mode: synchronous (commit waits for remote ack) vs local async (commit returns immediately).

**Added commit latency model (synchronous)**- Extra commit latency ≈ one RTT for leader→replica and ack back (often single RTT if pipelined):

text

Lcommit_sync = Llocal + RTT

- If durable fsync required on both sides before ack, include remote fsync Tr:

text

Lcommit_sync = Llocal + RTT + Tr

Plain-English: each commit waits for network roundtrip plus remote durability cost.

**Throughput impact**- Sync reduces max throughput because per-transaction critical path increases; throughput ≈ 1 / Lcommit_sync per single-threaded partition.- With concurrency C and contention, blocking increases tail latency; use:

text

Throughput ≈ min(C / Lcommit_sync, B / S)

where S = avg bytes/tx; bandwidth B caps if writes large.

**Bandwidth / cost estimate**- Bandwidth per second ≈ TPS * S * replication factor.- Monthly cost ≈ Bandwidth_per_sec * seconds_per_month * egress_price.Example: 10k TPS * 2KB * 3600*24*30 ≈ 518 GB/day → ~15.5 TB/month egress.

**Trade-offs**- Synchronous: strong consistency, higher latency, lower throughput, high egress cost; simpler conflict management.- Local async: low latency, high throughput, cheaper; risk of data loss and complex conflict resolution.

**Mitigations**- Batching: aggregate N txns into one replication message reduces RTT amortization.

text

Effective_RTT_per_tx ≈ RTT / N
Bandwidth unchanged but overhead reduced.

- WAN acceleration: TCP tuning, persistent connections, compression, UDP-based replication (e.g., QUIC), and proxies to reduce RTT and packet loss effects.- Semi-sync/hybrid: leader waits for durable ack from local sync replica and only asynchronously to remote region; or wait for async buffer flush remotely within a bounded time (SLO-based).- Sharding/affinity: route writes to nearest region owners to avoid cross-region sync.- Backpressure & adaptive throttling: drop or delay noncritical writes to protect tail latency.

**Recommendation**- For strict RPO=0 choose synchronous with WAN accel and batching; for typical business SLAs choose hybrid semi-sync (local sync + async cross-region) and async fallback with disaster recovery playbooks. Quantify per-app using the latency model above and run experiments under real RTT and loss profiles.

EasySystem Design

30 practiced

Describe a DR runbook template for a single mission-critical service. Provide key fields that should be present such as triggers, preconditions, exact recovery steps (including IaC commands if applicable), verification checks, rollback criteria, stakeholders to notify, and estimated timelines for each major step.

Sample Answer

**Situation / Purpose**A concise DR runbook template for one mission‑critical service (e.g., customer-auth API) that a Cloud Architect would produce to ensure predictable recovery.

**Trigger / Activation**- Primary triggers: region outage, RTO breach, data corruption, failed failover tests.- Authority to declare: SRE lead or on‑call manager.

**Preconditions**- Last successful backup timestamp + retention window- Active IaC repo and pipeline accessible- Access: cloud admin, CI/CD token, DB restore creds- Health checks and monitoring endpoints defined

**Exact Recovery Steps**1. Declare incident and create incident channel (PagerDuty, Slack).2. Failover DNS / Traffic: update DNS weight or promote secondary region.3. Recreate infra (example Terraform):

bash

# switch to DR workspace and apply
terraform workspace select dr || terraform workspace new dr
terraform apply -var-file=dr.tfvars -auto-approve

4. Restore data:

bash

# restore DB snapshot (AWS example)
aws rds restore-db-instance-from-db-snapshot --db-instance-identifier auth-dr --db-snapshot-identifier snapshot-2026-03-01

5. Deploy service:

bash

# trigger pipeline
curl -X POST https://ci.example.com/builds -d '{"repo":"auth-service","ref":"release"}'

**Verification Checks**- Endpoint smoke: GET /health returns 200 in <2s- Authentication flow test with synthetic user- DB consistency checks (row counts, checksums)- Load test at 10% traffic ramp for 15 min

**Rollback Criteria**- Errors in >20% smoke checks, DB restore failure, latency >200ms for 10m- Rollback actions: re-point traffic back, destroy DR infra:

bash

terraform destroy -var-file=dr.tfvars -auto-approve

**Stakeholders & Notifications**- SRE Lead (primary), Cloud Architect, Product Owner, Security, Legal, Exec on-call.- Communication cadence: initial, hourly, post‑mortem.

**Estimated Timelines**- Detection → declare: 5–15 min- Infra provision: 15–30 min- DB restore: 30–90 min (depends on size)- Deploy & verification: 15–30 min- Full recovery target (RTO): 60–180 min

**Notes / Post‑Incident**- Capture root cause, update runbook with lessons learned, run tabletop within 7 days.

EasyTechnical

21 practiced

As the Cloud Architect, describe your immediate set of responsibilities and actions during a multi-region major incident that impacts production services. Include technical triage tasks, leadership and communication (to execs and customers), coordination with SRE, security, and legal teams, and which decisions should be deferred until the incident is stabilized.

Unlock Full Question Bank

Get access to hundreds of Disaster Recovery and Business Continuity interview questions and detailed answers.

Join thousands of developers preparing for their dream job.