Backup and Disaster Recovery Questions
Keeping data durable and recoverable when systems fail: backup design (full, incremental, differential, and snapshot strategies; point-in-time recovery), backup verification and restore testing, retention and archival policy (including compliance retention and legal holds), encryption and key management for backups, and disaster-recovery planning measured against recovery-time and recovery-point objectives (RTO/RPO). Tests whether a candidate can design a backup strategy that actually restores, choose the right retention tier for a business's downtime and data-loss tolerance, and operate backup systems safely under failure, compliance, and ransomware threats. Distinct from code-level fault-tolerance patterns (circuit breakers, retries, bulkheads) and multi-region failover architecture, which belong to high-availability-and-disaster-recovery.
Case study: A compliance workload requires 10-year retention and must be recoverable within 48 hours. Compare two approaches quantitatively and qualitatively: (A) keep replicated backup copies in two regions (higher ongoing storage/egress cost) vs (B) keep primary in lower-cost cold archive and perform on-demand restores to recover within 48 hours. What variables determine the better approach and how would you model costs and risk?
Sample Answer
Direct answer
Build a total-cost model over the same ten-year horizon for both approaches, using the same
dataset size on both sides, then find the break-even restore frequency at which the two costs are
equal. In a realistic model, keeping a cold archive copy and restoring on demand is dramatically
cheaper unless full-dataset restores happen roughly monthly or more often, which is far more often
than a genuine compliance archive is typically accessed. The variable that actually decides the
question is expected restore frequency, not raw storage price, though a hard regulatory
requirement for geographic redundancy can override the cost comparison entirely.
Quantitative model
Take an illustrative 8,000 GB (8 TB) compliance dataset, retained for 10 years, with a 48-hour
recovery requirement. All figures below are illustrative order-of-magnitude prices for
demonstrating the model, not live vendor pricing, and every step stays on the same basis: GB,
GB-month, and dollars per year, converted consistently.
Approach A: replicated copies in two regions, on a tier that supports near-immediate restore,
at an illustrative $0.01 per GB-month per region.
- Per region: 8,000 GB times $0.01 per GB-month is $80 per month.
- Two regions combined: $160 per month, or $1,920 per year, or $19,200 over 10 years.
- Plus a one-time cross-region seed transfer of roughly 8,000 GB at an illustrative $0.02 per GB
egress rate: about $160. - Total over 10 years: approximately $19,360.
Approach B: a single-region cold archive tier plus on-demand restores, at an illustrative
$0.001 per GB-month for storage and $0.02 per GB for a retrieval that still meets the 48-hour
target.
- Storage: 8,000 GB times $0.001 per GB-month is $8 per month, or $96 per year, or $960 over 10
years. - Retrieval: 8,000 GB times $0.02 per GB is $160 per full-dataset restore event.
Break-even. Setting the two ten-year totals equal: 960 plus 160 times N equals 19,360, so N
equals about 115 restore events over 10 years, roughly 11.5 per year, close to monthly. Below that
frequency, approach B is cheaper; above it, approach A is cheaper. At a more realistic frequency
for a legal-hold or audit archive, say 3 full-dataset restores across the entire 10-year hold,
approach B totals 960 plus 3 times 160, or $1,440, versus approach A's $19,360, roughly 13 times
cheaper.
Qualitative factors
- SLA margin, not just average cost. Archive tiers commonly offer a faster, pricier retrieval
option and a slower, cheaper one. For a 48-hour requirement, choose whichever tier's stated
retrieval window leaves real margin under 48 hours, not one whose window is itself close to 48
hours, since that leaves no time for locating the right data, transferring it, and validating it
before the deadline. - Regulatory mandates can override cost. Some compliance regimes require genuinely
geographically redundant, continuously available copies as a requirement in itself, independent
of how often the data is actually restored. Check whether that applies before running any cost
model; if it does, approach A may be mandated regardless of the comparison above. - Tail risk of a single archive copy. If approach B's single region is lost entirely before a
restore happens, and the archive tier does not itself provide independent geographic redundancy
beyond its own region, approach B carries more risk of total data loss than approach A's
two-region design. This risk does not show up in the average-cost model and has to be weighed
separately.
Variables that determine the better approach
Dataset size, retention duration, the price differential between the fast and cold tiers, expected
restore frequency over the retention horizon (the single largest lever, per the break-even
calculation), the archive tier's actual retrieval latency relative to the SLA's margin, and any
hard regulatory requirement for geographic redundancy that removes the choice from a pure cost
comparison.
How to model cost and risk
Compute storage cost as footprint in GB times price per GB-month times months retained, summed per
copy, and retrieval cost as expected number of restores times footprint times price per GB
retrieved, comparing both approaches over the identical retention horizon and dataset size so
nothing is compared across mismatched bases. Score the qualitative risks, SLA margin and single-copy
exposure, separately from the dollar figures, and check whether a hard compliance requirement
removes the choice before optimizing the remaining trade-off on cost alone.
Design a backup and retention plan for a three-tier web application: stateless frontend web servers, application servers, a PostgreSQL primary with replicas, and S3-like object storage for user uploads. Business requirements: web/app RTO 2 hours, DB RPO 15 minutes, retention: 30 days for user uploads hot tier and 1 year for transactions. Describe backup types, cadence, storage tiers, and recovery order.
Sample Answer
Direct answer
Treat the two RTO/RPO numbers as two different engineering problems, not one: the 2-hour RTO for the stateless web/app tier is an infrastructure-provisioning problem solved with immutable images and IaC (no backup of the tier itself is needed, only of its definition), while the 15-minute RPO for PostgreSQL is a data-continuity problem that a nightly or even hourly backup cannot satisfy on its own and requires continuous WAL (write-ahead log) archiving or streaming replication. Recovery order follows the dependency graph: infrastructure and the database restore in parallel where possible, but the app tier cannot serve correct traffic until the database is verified, so the database restore is the critical path inside the 2-hour budget, not an independent 2-hour budget of its own.
Backup types by component
- Frontend and app servers (stateless): no data backup at all. "Backup" here means an immutable machine image or container image plus the Infrastructure-as-Code (IaC) that defines how many instances, what config, and what network wiring, stored in version control. Recovery is redeployment, not restore.
- PostgreSQL primary with replicas: a weekly or nightly full base backup, plus continuous WAL archiving (shipping WAL segments to durable storage as they are generated, not on a fixed interval) to support point-in-time recovery (PITR). Separately, keep at least one warm standby replica for fast failover, since PITR from a cold base backup is a different, slower recovery path than promoting an already-current replica.
- S3-like object storage for user uploads: object storage is typically already durable (multi-AZ replicated by the provider), so "backup" here means versioning plus cross-region replication to protect against accidental deletion, overwrite, or a regional failure, not protection against media failure.
Same-basis check on the RPO number
15 minutes RPO means: after any failure, at most 15 minutes of committed transactions may be unrecoverable. A nightly full backup alone gives an RPO of up to 24 hours, which does not meet this requirement by two orders of magnitude, so cadence has to be re-derived from the RPO target, not assumed. Continuous WAL archiving with a shipping interval well under 15 minutes (WAL segments are typically archived every few seconds to a couple of minutes under normal load) comfortably meets the target with margin; if only a fixed-interval WAL push is available rather than continuous streaming, set that interval to something like 5 minutes, not 15, because the RPO has to account for the interval plus shipping latency plus detection time, and setting the interval exactly at the SLA boundary leaves no margin for any of those.
Storage tiers
- User uploads, 30-day hot tier: keep the most recent 30 days in a standard, low-latency object storage class since that is the window most likely to be accessed or need a fast individual-object restore (accidental deletion, corruption). Interpreting the requirement as "uploads are retained in the hot tier for 30 days" (rather than "uploads are deleted after 30 days"), lifecycle older objects to a cheaper infrequent-access or archive tier rather than deleting them, unless the business has separately confirmed 30 days is the full retention period; this is stated as an assumption because the question does not specify what happens to uploads after day 30.
- Transactions, 1-year retention: recent PostgreSQL backups (say the last 1-2 weeks of base backups plus WAL) stay in fast storage for quick PITR; older backups within the 1-year window move to a cheaper archive tier where restore latency of hours is acceptable, because a 300-day-old backup is being kept for audit or compliance reasons, not for a 2-hour RTO scenario.
Recovery order and a worked RTO check
- Provision network and load balancer baseline via IaC. This has no data dependency and can start immediately, in parallel with step 2.
- Restore the PostgreSQL primary (the critical path): promote an existing warm standby if one survived the failure (fast, typically single-digit minutes), or restore the latest base backup and replay WAL to the most recent consistent point if the standby is also lost (slower, and must be timed).
- Verify database integrity (checksums, row counts, application-level sanity queries) before pointing anything at it.
- Deploy app servers from the pinned image/IaC pointed at the verified database; this is typically fast (minutes) since it is just infrastructure provisioning with no data to move.
- Deploy frontend web servers, same mechanism.
- Warm any caches, then cut traffic over.
Worked check on whether cold restore fits the 2-hour budget: suppose the database is 500 GB and cold restore-plus-replay throughput from the backup store is, as an illustrative ESTIMATE, 500 MB per minute. 500000/500=1000 minutes, about 16.7 hours, which massively breaches the 2-hour RTO even before the stateless tiers are provisioned. This is the concrete reason a cold base-backup restore should not be the primary recovery path for the app tier's 2-hour RTO: keep a warm standby that can be promoted in minutes as the primary recovery mechanism, and treat cold PITR restore as the fallback for a failure mode a standby cannot cover, such as logical corruption replicated to the standby, where the RTO target may need to be renegotiated with the business since a large cold restore genuinely cannot fit inside 2 hours at typical restore throughput.
Where the numbers do not directly compare
The RTO (2 hours, for web/app) and RPO (15 minutes, for the database) are not the same measurement and should not be added or averaged: RTO is "how long until service is back," RPO is "how much data could be lost." A design can meet both independently (standby promotion gives a low RTO; continuous WAL archiving gives a low RPO) but meeting one does not imply anything about the other, and a plan that only discusses one while citing both numbers has not actually answered the question.
Design a disaster recovery strategy for a transactional relational database requiring RPO < 1 minute and RTO < 15 minutes across regions. Describe replication method, failover procedure, data consistency considerations, and how to test DR with minimal production impact.
Sample Answer
Direct answer
RPO (Recovery Point Objective: the maximum data-loss window) under 1 minute is tight enough to change the design decision compared to a looser target: at this ceiling, asynchronous replication's typical seconds-level lag is often fine in practice, but it isn't a guaranteed bound, replication lag can spike well past a minute under load or a network hiccup with nothing stopping it. Meeting RPO < 1 minute as an actual guarantee, not just a usual outcome, points toward synchronous or quorum-based replication (the primary waits for acknowledgment from the standby, or from a quorum of standbys, before committing), accepting the added write latency that comes with it as the deliberate cost of the guarantee.
Structured elaboration
Replication method. Synchronous replication (e.g. Postgres synchronous_commit=on or remote_apply) gives RPO effectively zero for infrastructure failures, since no commit is acknowledged to the client until the standby has it, but adds the cross-region round-trip time (commonly tens to over a hundred milliseconds depending on the region pair) to every write, which can become a real throughput constraint under high transaction rates. Quorum-based replication (a write is acknowledged once a quorum, e.g. 2 of 3 regional replicas, has it, not all of them) gives the same strong guarantee while tolerating one slow or unreachable replica without stalling the primary. A well-monitored asynchronous setup, with alerting on lag well inside the 1-minute budget, is a real option only if the business explicitly accepts "usually under a minute, not contractually guaranteed under adverse conditions"; that trade-off should be surfaced explicitly rather than silently assumed away by picking async and hoping.
Failover procedure and RTO budget. Detection (confirmed failure, not a single blip): roughly 2 minutes. Promotion: roughly 3 minutes, faster than a looser-RPO design because a synchronous or quorum replica is, by construction, already caught up and doesn't need catch-up time before it's safe to promote. Traffic cutover: roughly 5 minutes, requiring a short DNS TTL (time-to-live, how long clients cache a DNS record before re-checking it) set in advance (e.g. 30 to 60 seconds) plus application-level reconnect/retry behavior that doesn't require a manual restart. Validation smoke test before declaring recovery: roughly 3 minutes. Summed: roughly 13 minutes, leaving about 2 minutes of margin under the 15-minute RTO (Recovery Time Objective: the maximum downtime) target, all figures illustrative and meant to be re-timed in a real drill, not asserted as guaranteed.
Data consistency considerations. With synchronous or quorum replication, the promoted standby is guaranteed consistent up to the last acknowledged commit, so failover requires no reconciliation, the biggest consistency advantage over asynchronous replication. The real risk with any replication mode is the old primary coming back online after failover with writes the new primary never received (a split-brain divergence): it has to be explicitly fenced (its ability to accept writes revoked) the moment failover is declared, and any writes it accepted after the split should be treated as lost or requiring manual reconciliation, not silently merged back in.
Testing DR with minimal production impact. Promote a replica into an isolated network segment for drills, not the live traffic path, so testing the mechanics of failover never touches real user traffic. Validate the promoted instance with synthetic or replayed read-only traffic, or a canary write workload against a namespaced test schema, rather than pointing real users at it. To test the cutover mechanism itself (DNS or routing changes) without customer impact, use a small canary percentage of traffic, or hit the DR region's endpoint directly with an internal synthetic client, bypassing the global router that serves real users, so the target environment is validated without touching what actually routes production. Time every phase of each drill against the 15-minute budget and update the runbook if any phase trends over its allotted share.
Design a highly available PostgreSQL-based OLTP architecture on AWS with a 99.99% availability target. Would you reach for RDS Multi-AZ or Aurora, and why? Walk through the rest of your design and how each choice trades off against RTO/RPO and cost.
Sample Answer
Direct answer
Both RDS Multi-AZ and Aurora give synchronous, multi-AZ data durability with an RPO (Recovery Point Objective, maximum acceptable data loss) effectively at zero for infrastructure failures within a region, so the 99.99% availability target is achievable with either. The differentiator is failover speed and cross-region disaster recovery: Aurora's storage layer is already replicated across AZs independently of how many compute replicas you run, which typically makes its failover faster than RDS Multi-AZ's, and Aurora Global Database gives materially tighter cross-region RPO than a standard RDS cross-region read replica, at a real added cost. Neither option, on its own, protects against a bad write or a human error like an unqualified DELETE, since synchronous replication faithfully replicates mistakes just as reliably as it replicates good writes; that's what point-in-time recovery (PITR) is separately for.
Structured elaboration
RDS Multi-AZ. A standard managed Postgres instance with a synchronous physical standby in a second AZ (Availability Zone: an independent data center within the same region) in the same region. Failover is automatic on primary failure, typically completing in roughly one to two minutes per AWS's own documentation (a vendor-reported figure, not independently measured here, and worth treating as approximate), via a DNS endpoint flip to the new primary. RPO is effectively zero for AZ-level or instance-level failures, since the standby only acknowledges synchronously. This covers AZ and instance failure within one region; it does not cover a full regional outage or a logical error, both of which need a separate mechanism: automated backups plus continuous WAL (write-ahead log) archiving to object storage for PITR, and optionally an asynchronous cross-region read replica that can be promoted for regional DR, with RPO in that case bounded by replica lag rather than zero.
Aurora. A Postgres-compatible managed service where the underlying storage layer is itself distributed and replicated across multiple AZs (commonly reported as 6 copies across 3 AZs) independent of how many compute instances are provisioned, so a single Aurora instance already has multi-AZ storage durability built in. Failover to an Aurora Replica, where one exists, is typically faster than RDS Multi-AZ's, often reported in the tens of seconds, because the new primary doesn't need to redo as much work on promotion, the storage layer is already shared and current (again a vendor-reported figure, flagged as such rather than personally benchmarked). RPO for AZ-level failure is effectively zero for the same reason. Aurora Global Database extends this with asynchronous cross-region replication, commonly reported at sub-second to low-single-digit-second lag, materially tighter than a standard RDS cross-region replica's typical lag (also vendor-reported, not independently verified here).
Recommendation and the RTO/RPO/cost trade-off. If the 99.99% target is purely about tolerating an AZ-level failure within one region, plain RDS Multi-AZ is the more cost-effective choice: it clears that bar with a simpler pricing model (roughly instance-plus-storage cost doubled for the standby) and doesn't require paying for Aurora's more sophisticated storage layer or a Global Database second region you may not need. If regional disaster recovery is also a genuine requirement, worth calling out as a distinct requirement from the AZ-level 99.99% figure, not something automatically implied by it, Aurora Global Database earns its added cost through a materially tighter cross-region RPO than a standard RDS cross-region replica provides.
The rest of the design. Layer PITR on top of whichever option is chosen: both support continuous backup to object storage with restore to any specific second within the retention window (commonly up to 35 days), which is the actual safety net for logical or human error, something no amount of synchronous replication provides. Run periodic DR drills that promote a cross-region replica into an isolated network and validate the real, measured regional RTO and RPO against target, the same rehearsal discipline any DR design needs, rather than trusting the vendor-documented failover numbers without ever having exercised them for real.
Design a pattern for backing up and restoring petabytes of object storage data across regions, minimizing transfer costs while still meeting a 24-hour restore SLA for your most critical objects. How would you make a selective restore of just the objects you need fast, without restoring everything?
Sample Answer
Direct answer
The pattern that satisfies both constraints, transfer cost and a 24-hour restore SLA for the most critical objects, is tiering by criticality rather than replicating everything uniformly, backed by an independent, always-available metadata catalog that lets you resolve exactly which objects an incident requires without scanning the whole store. Cost scales with what you actually replicate and at what storage tier, not with the size of the total archive, which is the lever that makes petabyte-scale DR affordable at all.
Structured elaboration
Minimizing transfer cost. Most object stores at this scale have a long tail: a small fraction of objects account for most access, and a large share of the rest is genuinely cold. Replicate incrementally at the object level, only new or changed objects (detectable via version ID, ETag, a hash-based fingerprint of the object's contents that changes whenever the object does, or last-modified timestamp, avoiding re-transferring unchanged data), and place the replicated copies of cold objects in a cheaper archive storage class in the DR region rather than a uniformly warm one. That lowers ongoing storage cost even though it raises the retrieval latency for those specific objects, which is acceptable because the 24-hour SLA is explicitly scoped to the most critical objects, not the whole archive.
Worked cost example (basis: the size of the replicated critical subset, not the total store). Suppose the critical subset is 10 TB out of a 5 PB total store. Replicating just that 10 TB continuously costs egress on 10 TB, not on 5,000 TB: at an illustrative $0.02/GB egress rate, that's 10,000 GB x $0.02 = about $200 one-time (plus small ongoing costs for deltas), versus replicating the full 5 PB at the same rate, 5,000,000 GB x $0.02 = about $100,000, roughly a 500x difference. This is exactly why uniform replication doesn't scale at petabyte size and tiering by criticality does.
Meeting the 24-hour restore SLA for critical objects specifically. Keep the critical subset in a warm storage class in the DR region, immediately readable with no thaw or retrieval delay, so its restore time is bounded by network transfer and API call time, not by an archive-retrieval wait. This matters because some archive storage classes have retrieval delays ranging from minutes to tens of hours depending on the tier chosen; placing critical objects in a deep-archive tier by mistake could consume the entire 24-hour SLA before any data even starts moving, independent of network speed. The storage class chosen for the critical subset is itself an SLA-determining decision, separate from and at least as important as network throughput.
Selective fast restore without restoring everything. This requires a metadata catalog, an index of object key, version, checksum, storage tier and location, backup timestamp, and criticality classification, maintained independently as a small, always-hot database, not embedded inside the bulk archive itself (the same principle as keeping a backup catalog out of the data center it describes: if the catalog only lives inside the store that just had an incident, you can't find anything to restore). The catalog resolves "which objects does this incident actually require" to a precise list quickly, and that list drives targeted, parallelized restore requests for exactly those objects, since object stores support high per-object parallelism, rather than the much slower default of restoring a whole time-ordered prefix and scanning through it to find what's needed.
Putting it together. The pattern is: replicate incrementally and selectively rather than uniformly; place the critical subset in a warm, immediately-readable tier and everything else in cheaper archive tiers; maintain an independent, fast metadata catalog that turns "restore what we need" into a precise, parallel operation instead of a full-archive scan. Validate the design by periodically restoring a sample from the critical tier and timing it against the 24-hour SLA, and separately confirming the catalog itself stays available and accurate even during a regional incident.
Unlock Full Question Bank
Get access to all 11 Backup and Disaster Recovery interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.