Backup and Disaster Recovery Questions

Direct answer

When a petabyte-scale WAN backup keeps blowing past its window, the fix that matters most is almost never "add more bandwidth" or "compress harder": it's turning the backup from a repeated full transfer into a deduplicated, incremental one, because that changes the amount of data crossing the WAN by an order of magnitude, while every other lever (compression, parallel streams, scheduling) only shaves a constant factor off whatever volume you chose to send.

Structured elaboration

Global/source-side deduplication. Deduplicating at the source, before data crosses the WAN, means only genuinely unique bytes are ever transmitted; blocks identical to something already backed up (common across similar files, repeated database pages, near-duplicate snapshots) are represented by a small reference instead of being resent. This is the single highest-leverage change for a recurring, blown-window backup because it attacks the actual bottleneck (bytes over a bandwidth-limited link), not the transfer mechanism.

Block-level incremental (delta) transfer. Once a full baseline exists, every subsequent run should only ship the blocks that changed since the last run, not a fresh full copy of everything. Combined with dedup, this is what turns a petabyte-scale nightly job into a job proportional to that day's actual churn rate rather than the total dataset size.

Physical/offline seeding for the initial baseline. The very first full transfer of a petabyte-scale dataset over a bandwidth-limited WAN link is often infeasible to complete in any reasonable window at all (the worked example below shows why); shipping the initial baseline via a physical storage transfer to the target site, then switching to incremental WAN transfer for everything after that, avoids trying to solve a one-time problem with a permanent-capacity fix.

Compression. Standard compression typically yields a real but modest reduction (often in the range of two to three times smaller, heavily dependent on how compressible the data already is), which helps but does not by itself close a large window overrun the way volume reduction from dedup and incrementals does.

Bandwidth shaping, scheduling, and parallelism. Running more concurrent streams and scheduling the job to avoid competing with other WAN traffic can use the available link more fully, but these levers are capped by the physical link's total capacity; they help you use 100% of what you have, they cannot manufacture bandwidth you don't have the way reducing the bytes you need to send does.

Which change matters most. For a backup that has been consistently missing its window (not a one-off anomaly), the dominant fix is deduplication plus incremental transfer, because it reduces the transferred volume itself; compression, parallelism, and scheduling are worth doing too, but they're secondary because they optimize how efficiently you move whatever volume you decided to send, not how much volume that is.

Worked example

Take a 2 PB dataset (using decimal units throughout for a consistent basis: 1 GB = 10^9 bytes, 1 PB = 1,000,000 GB) over a 10 Gbps WAN link. Converting the link speed to the same byte basis: 10 Gbps = 10 x 10^9 bits per second / 8 = 1.25 GB/s. A full transfer of the entire 2,000,000 GB dataset at that rate takes 2,000,000 / 1.25 = 1,600,000 seconds, about 444 hours, about 18.5 days: nowhere close to any nightly or even weekly window, and this is before accounting for the fact you'd have to redo this every single run if you kept sending full copies. Now suppose the dataset has a 1% daily change rate. An incremental, deduplicated run only needs to send the changed 1% = 20,000 GB. At the same 1.25 GB/s, that takes 20,000 / 1.25 = 16,000 seconds, about 4.4 hours: comfortably inside a typical overnight window. The volume reduction from moving to incrementals (from 2,000,000 GB down to 20,000 GB, a 100x reduction in what needs to cross the WAN) dwarfs what compression (roughly 2-3x) or added parallel streams (bounded by the same 1.25 GB/s link ceiling regardless of how many streams share it) could deliver on their own.

Trade-offs and pitfalls

• Deduplication and delta transfer require tracking state (what has already been sent, what changed since last run) both at the source and target; that state itself needs to be backed up and verifiable, since a corrupted change-tracking index can silently cause an incremental backup to miss data.
• Physical seeding solves the one-time baseline problem but adds operational complexity (shipping and securely handling physical media or an appliance) and a fixed delay before the target site is even up to date with the source at seed time; it is worth it once for a petabyte-scale initial load, not as an ongoing substitute for network transfer.
• Comparing "backup window duration" against "bandwidth" only makes sense once both are expressed on the same time basis and the same byte basis; a plan that quotes the link in bits per second and the dataset in bytes without converting between them (as this worked example deliberately does) is a common source of an order-of-magnitude planning error.

Direct answer

The design splits into two layers that solve different problems: continuous replication (streaming or WAL shipping, WAL being the write-ahead log, an append-only record of every change Postgres makes before applying it) gets you a warm standby you can promote quickly, meeting RTO; and independent base backups plus archived WAL get you point-in-time recovery (PITR) and protection against the kind of failure replication doesn't cover, like a bad DROP TABLE that a synchronous or asynchronous standby would faithfully replicate right along with the primary. For illustration I'll target RPO (Recovery Point Objective, the maximum acceptable data loss measured in time) of roughly 30 seconds and RTO (Recovery Time Objective, the maximum acceptable downtime) of roughly 15 minutes, both stated as example targets since the question leaves them to the candidate, not as universal numbers.

Structured elaboration

RDS vs self-managed on EBS. On RDS (a managed Postgres service), automated backups continuously stream WAL to object storage, giving PITR out of the box with retention up to 35 days, and a cross-region read replica (asynchronous by default) can be promoted for regional DR. On self-managed Postgres on EBS (Elastic Block Store, cloud block storage volumes), the same capability has to be built: pg_basebackup for periodic base backups, continuous WAL archiving to object storage (commonly via wal-g or pgBackRest) for PITR, and physical streaming replication to a standby in another region. EBS snapshots can also back the base-backup layer, but need care: a snapshot taken mid-write without coordinating with Postgres (e.g. via pg_start_backup) can capture an inconsistent volume state, so either use a backup tool that handles this coordination or take the base backup through pg_basebackup instead of a raw disk snapshot.

Cross-region replication: sync vs async. Synchronous replication (the primary waits for the standby to acknowledge before committing) gives an RPO close to zero but adds the cross-region round-trip time to every write's latency, commonly tens to well over a hundred milliseconds depending on the region pair, and risks stalling the primary if the standby becomes unreachable. Asynchronous replication accepts a small replication lag (commonly seconds under normal load) in exchange for no write-latency tax; given a 30-second RPO target, async has ample headroom and is the more cost-effective default. Reserve synchronous replication for a stricter RPO tier if one is ever needed.

Failover procedure. Promote the standby (pg_ctl promote, or the managed failover API on RDS), update the client-facing endpoint (DNS or a connection proxy) to point at the new primary, and fence the old primary, revoking its ability to accept writes, so it can't diverge from the newly promoted primary if it later comes back online (a split-brain scenario, where two nodes both believe they're primary and accept conflicting writes).

Worked RTO breakdown (basis: wall-clock minutes from declared incident to first successful write, summed and checked against the 15-minute target, not each phase in isolation). Detection and decision to fail over: about 5 minutes (health-check failure threshold plus confirmation). Promotion of the standby: about 2 to 5 minutes. Endpoint cutover and client reconnect: about 5 minutes (DNS propagation and application retry/reconnect behavior). Smoke validation before declaring the incident resolved: about 3 to 5 minutes. Summed, that's roughly 15 to 20 minutes; if the total lands over the 15-minute target, the endpoint-cutover step (usually the least automatable) is the first place to look for savings, e.g. via a connection proxy that can be repointed faster than DNS TTL propagation allows.

Validation and automated DR testing. Run scheduled drills that promote the DR replica into an isolated network (not the live traffic path), run automated read/write smoke tests and a row-count or checksum comparison against the primary before the drill promotes, and time every phase against the RTO/RPO targets. Separately, test the PITR path itself (restore a base backup plus archived WAL to a specific past timestamp) on its own schedule, since it's a different recovery mechanism from replica promotion and can silently break (a missing WAL segment, an expired retention window) without the replication drills ever exercising it.

Controlled restore of production traffic. After promoting the DR replica and validating it, don't cut 100% of traffic over immediately: shift a small percentage first via weighted routing, watch error rates and replication lag on the newly promoted primary, then ramp to full traffic. Keep the old primary around, fenced but not destroyed, as a forensic copy in case anything needs to be recovered from it, and rebuild a fresh standby in a healthy region afterward so the system isn't left without DR coverage once the immediate incident is over.

Direct answer

The single decision that determines everything else about a backup type is what it actually captures: a full backup captures every byte, an incremental backup captures only what changed since the last backup of any kind, and a differential backup captures everything that changed since the last full backup (ignoring any differentials in between). That one difference cascades into how restore works, how much storage and I/O each consumes, how long and fragile the recovery chain gets, and which scenario each fits. Point-in-time recovery via transaction-log shipping is a related but separate mechanism and isn't one of the three types asked about here.

Structured elaboration

Full backups.

• What it captures: a complete copy of all data at the moment the backup runs.
• Restore mechanics: apply a single file. No assembly, no ordering to get right.
• Storage and I/O: highest of the three, every run duplicates the entire dataset; backup-time I/O is also the highest since the whole dataset is read and written every time.
• Recovery chain complexity: none, it's a chain of length one.
• Realistic scenario: as the periodic anchor point underneath a mixed strategy (a weekly full backing incremental or differential backups on the days in between), or as a one-off just before a risky operation like a schema migration, where you want the simplest possible restore path available if it goes wrong, not a clever one.

Incremental backups.

• What it captures: only the data changed since the immediately preceding backup, whatever type that was (the last full, or the last incremental).
• Restore mechanics: restore the last full, then apply every incremental since it, strictly in order. If backups were taken Mon (full), Tue, Wed, Thu, Fri, Sat, Sun (incrementals), restoring to Sunday means applying all six incrementals on top of the full in sequence.
• Storage and I/O: lowest per individual backup run, since only changed data is captured each time, so this is the cheapest option in steady-state storage and the fastest to actually take.
• Recovery chain complexity: the highest of the three, and it grows every day since the last full. A single corrupted or missing incremental in the middle of that chain breaks the restore for every day after it, not just that one day.
• Realistic scenario: very large datasets with a tight backup window and a real storage-cost driver, e.g. a multi-terabyte warehouse where a nightly full backup wouldn't finish before the maintenance window closes and daily full copies would be prohibitively expensive to store, and where actual restores are rare enough that a longer, multi-file restore assembly is an acceptable trade for the ongoing savings.

Differential backups.

• What it captures: everything changed since the last full backup (not since the last differential), so each day's differential is a superset of the previous day's.
• Restore mechanics: restore the last full, then apply only the single most recent differential. Restoring to Sunday in the same Mon-Sun example above means applying the full plus only Sunday's differential, the Tue through Sat differentals are never needed.
• Storage and I/O: between full and incremental, and it grows every day since the last full as more cumulative change gets folded into each new differential; by the day before the next full, the differential can approach the size of a full backup of just the changed rows.
• Recovery chain complexity: fixed at two, full plus latest differential, regardless of how many days have passed since the last full. Losing an older differential doesn't matter because only the newest one is ever used.
• Realistic scenario: systems where restore speed and restore reliability matter more than minimizing backup-time storage, e.g. a transactional system with a demanding recovery-time target where you don't want six independent incremental files all needing to be intact to get back online; you're trading a bit more storage for a restore path that can't be broken by losing an old file.

Worked example, basis of every number stated. Assume a 500 GB database with a constant 10 GB/day of changed data (this is the daily delta measured in changed bytes, treated as constant for illustration; a real system's WAL (write-ahead log, the database's own transaction log) volume can exceed the net-changed-bytes figure because of engine-level effects like full-page writes (the database re-logging an entire disk page rather than just the changed bytes, the first time that page changes after a checkpoint), so treat 10 GB/day as the backup-relevant change volume, not raw log volume). A full backup runs Monday: 500 GB. Over the following six days:

• Incremental: each day backs up only that day's 10 GB, so Tue through Sun each write a 10 GB file, roughly 60 GB total across the week. Restoring to Sunday means applying the 500 GB full plus six 10 GB incrementals in order: seven files, seven steps.
• Differential: each day backs up everything since Monday's full, so the differentials are 10 GB (Tue), 20 GB (Wed), 30 GB (Thu), 40 GB (Fri), 50 GB (Sat), and 60 GB (Sun), since each day rolls in one more day's worth of change on top of the same base. Restoring to Sunday means applying the 500 GB full plus only the 60 GB Sunday differential: two files, two steps, even though the Sunday differential file itself is as large as the entire incremental week combined.

That last line is the trade-off in one sentence: incremental keeps each individual backup small at the cost of a restore chain that gets longer and more fragile every day; differential keeps the restore chain fixed at two files at the cost of each individual backup getting larger every day since the last full.

Direct answer

A backup job reporting success typically only proves the process ran to completion without an
unhandled error. It says nothing about whether the resulting data is complete, uncorrupted, or
usable in a real restore. Confidence has to come from evidence that recovery actually works, built
both as a right-now check and as an ongoing habit, not from trusting the job log.

Structured elaboration

What would make you confident right now.

• Checksum or hash verification. Confirm the checksum of the data as written matches the
  checksum of the data as read from source. This catches corruption in transit or at rest, but not
  logical or application-level correctness.
• Catalog completeness. Confirm the backup catalog lists the expected number of files, objects,
  or databases, not a partial subset. This is what catches a job that technically succeeded but
  only captured 80% of the source because of a permissions error on one folder that got logged as a
  warning rather than a failure, which the daily job-status review would need to be looking for
  specifically.
• An actual restore into an isolated, scratch environment, followed by an application-level
  check: mount the filesystem, attach the disk, open the database and run a query, or boot the
  virtual machine. This is the only step that tests the entire chain at once, the backup software,
  the media, the restore procedure, and compatibility with the target environment, and it is the
  only one of the three that can catch a failure the other two would both pass.

How to build that confidence on an ongoing basis.

• Scheduled restore testing on a rotation, covering every system on a fixed cadence rather than
  testing once and assuming it still holds. Software versions, storage drivers, and licensing all
  drift over time, and a restore path that worked a year ago can silently break without any
  backup job ever reporting an error.
• Automated or synthetic verification where the backup product supports it: many can auto-mount
  a backup image in an isolated sandbox and run an automated health check without a human running a
  manual restore every time, making frequent, even nightly, verification affordable at scale. Treat
  this as a complement to periodic real restore drills, not a full replacement, since it typically
  validates the backup software's own read-back rather than a fully independent restore into a
  clean target.
• Track restore-test results as a trend, not a single pass or fail, so a slow drift, restore
  times creeping upward, or an increasing rate of "restore succeeded but the application check
  failed", becomes visible before it turns into a crisis.
• Trigger an out-of-cycle restore test on any material change: new backup software version,
  storage migration, or application upgrade, rather than waiting for the next scheduled test.

Worked scenario

During a scheduled monthly restore drill, a database backup mounts cleanly and its checksum
matches. The application-level check then fails: a stored procedure the application relies on
references a linked server that no longer exists in the restore target. The checksum was correct
and the mount succeeded; only the application-level check caught this. That is exactly the class of
failure a green backup job would never reveal, which is why "restorable" has to mean "passes an
application-level check", not just "the bytes match."

Trade-offs

Full restore drills cost real time and infrastructure, which is why many teams under-invest here.
Synthetic verification is a reasonable, cheaper middle layer for frequent coverage, but it should
never be the only check performed, since it usually cannot catch the kind of environment-specific
or dependency-level failure the worked scenario above describes.

Direct answer

Verifying 100 TB of mixed databases and file stores needs layered checks at different costs and frequencies: cheap, automated checks run on every backup every day, genuine full restores run on a rotating sample because exhaustively restoring everything daily isn't feasible at that scale, alerts are severity-aware rather than uniform, and the workflow explicitly separates a real backup failure from a false positive in the testing process itself before treating either as an incident.

Structured elaboration

Daily lightweight checks. Run cheap, fast checks against every backup every day: confirm the backup job actually completed successfully, sanity-check the size (a backup that's dramatically smaller than the same system's recent history often indicates a partial or silently-truncated backup that still reported success), and verify a checksum of the stored backup file itself against what was written. This catches storage-layer corruption and obviously broken backups without the cost of an actual restore, and it's cheap enough to run on the full 100 TB estate every single day.

Periodic full-restore tests. On a rotating schedule rather than daily for everything, since fully restoring 100 TB of mixed systems every day is not feasible within any reasonable time or cost budget, actually perform a real, end-to-end restore into an isolated environment and validate at the application level, not just confirm files landed. For a database that means running integrity checks, row counts, or a representative smoke query; for a file store it means confirming a sample of files are actually readable and intact. This is the only check in the whole process that catches "the backup reports success and passes its checksum but is not actually restorable," a materially different and more dangerous failure mode than simple corruption, since it hides behind every cheaper check passing.

Automated sampling. Given the infeasibility of testing everything at full-restore depth constantly, use a statistically-informed sampling strategy: weight selection toward higher-risk or higher-value systems (production databases ahead of low-priority development scratch space), and rotate the sample deliberately so that over a defined period, a quarter for example, every system in the estate has actually been full-restore tested at least once, rather than the sampling process repeatedly and conveniently re-testing the same easy, already-known-good subset.

Alerting on failures. Any daily-check failure or restore-test failure notifies the owning team, with urgency tied to what actually failed: a checksum mismatch on a critical production system's backup is a different severity than a slow restore-test result on a low-priority archival dataset. A daily-check failure should trigger an automatic re-check before escalating loudly, since a single transient issue (a brief network error during a checksum read) is a genuinely different situation from a persistent, repeatable failure, and treating both identically either causes alert fatigue from noisy transients or under-reacts to a real, recurring problem.

False positives and remediation. A real false-positive class exists here: a restore test can fail because the isolated test environment itself had a problem, insufficient capacity, a misconfiguration, not because the underlying backup was actually bad. The workflow needs to distinguish "the backup is genuinely bad" from "the test harness had a problem" before escalating as a confirmed incident, typically through an automatic retry in a known-clean environment before declaring a real failure. For a confirmed genuine failure, the remediation workflow should immediately trigger an out-of-band fresh backup of the affected system rather than waiting for its next regularly scheduled run, plus open a ticket to root-cause why the prior backup was actually bad, so the same underlying defect doesn't quietly recur on the very next cycle.

Worked example

A 100 TB mixed environment runs daily checksum and size-sanity checks across all systems, which catch a partial backup on a file server the same night it happens (its size came in at a fraction of its typical daily volume, triggering an automatic re-check and then an alert when the re-check confirmed the shortfall). Separately, a rotating full-restore-test schedule covers roughly 25% of systems each quarter, weighted so every production database gets tested at least once a quarter while lower-priority systems rotate through over a full year; this quarter's sample restore-tests a production database into an isolated environment and runs a row-count and smoke-query check. The first attempt fails, but before escalating, the automated workflow retries in a freshly-provisioned isolated environment and the second attempt passes cleanly, correctly identifying the first failure as a test-harness capacity issue (the isolated environment ran low on disk mid-restore) rather than an actual backup defect, avoiding an unnecessary incident and a false alarm to the owning team.

Trade-offs and pitfalls

• Sampling that isn't deliberately rotated tends to drift toward re-testing whatever's easiest or already known-good, which quietly leaves some systems untested for a full-restore check indefinitely; the rotation needs an explicit, tracked guarantee that every system gets covered within a bounded period.
• Treating every restore-test failure as a confirmed incident without a retry step in a clean environment first risks flooding teams with false alarms caused by the test harness itself, which erodes trust in the whole verification process over time.
• Daily checks alone, however consistently they pass, cannot catch "restorable in theory but broken in practice," which is exactly the gap periodic full-restore testing exists to close; relying on daily checks alone is a common, dangerous shortcut at this scale.