Backup and Disaster Recovery Questions

Direct answer

At 200 TB, two things break the naive reading of the requirements and have to be designed around explicitly. First, "weekly full backup" cannot mean literally re-reading and re-copying all 200 TB every week: at a realistic sustained backup throughput of 500 MB/s, copying 200 TB takes roughly 200,000,000 MB / 500 MB/s ≈ 400,000 seconds ≈ 111 hours, well over four days, so a literal weekly full would never finish before the next one starts. The fix is snapshot-based, copy-on-write "synthetic full" backups: a full recovery point that is constructed from an incremental chain without re-reading the whole dataset. Second, RPO < 1 hour (Recovery Point Objective: the maximum acceptable data loss, measured as time) is stated only for the highest-priority subset of data, not the whole 200 TB; daily incrementals alone give an RPO of up to 24 hours, so that highest-priority subset needs a separate, more frequent capture mechanism layered on top of the daily/weekly baseline, not a tighter version of the same schedule applied everywhere.

Structured elaboration

Architecture: snapshot vs block-level copy vs agent, and why a hybrid.

• Storage-level snapshots (copy-on-write snapshots at the SAN, Storage Area Network, a dedicated network of shared block-storage devices, array, or cloud block-storage layer) are fast to create, capture only changed blocks, and are the right mechanism for the frequent, high-priority tier: they don't require reading the whole volume, so an hourly snapshot on a 200 TB estate is cheap regardless of total size. They are typically crash-consistent by default (consistent with an ungraceful power-off) and need an application-aware quiesce step (flushing writes, briefly freezing the filesystem) to be application-consistent for databases.
• Block-level backup/replication (deduplicated, changed-block backup software, or asynchronous block replication to the second data center) is what actually gets data across data centers for disaster recovery, since a local snapshot alone doesn't survive losing the data center it lives in. This is the mechanism that should carry the highest-priority subset's near-real-time copy to the DR site.
• Agent-based, file-level backup (an agent running inside each host or VM that understands the application, e.g. taking an application-consistent dump via a pre-freeze hook) is the most flexible but the slowest to scan at this scale, and shouldn't be the primary mechanism for the bulk of 200 TB. Reserve it for stateful applications (databases, message queues) that need an application-consistent capture the block layer can't provide on its own, layered on top of block-level backup for the rest.
• Recommended hybrid: block-level, copy-on-write snapshots as the default for all volumes (daily, feeding synthetic weekly fulls); asynchronous block-level replication to the second data center for the highest-priority subset, running frequently enough (e.g. every 15 to 30 minutes) to comfortably clear the 1-hour RPO with margin; agent-based, application-consistent backups only for databases and similar stateful services that need a coordinated quiesce, at the same cadence as their tier.

Worked restore-time example (basis: a single critical dataset's volume, not the full 200 TB). Suppose the highest-priority tier is a 10 TB subset. At an aggregate parallel restore throughput of 1 GB/s (achievable by restoring many volumes or shards concurrently rather than serially), restoring 10 TB takes 10,000 GB / 1 GB/s = 10,000 s ≈ 2.8 hours, comfortably inside the 4-hour RTO (Recovery Time Objective: the maximum acceptable downtime) for that dataset. This number is scoped to the 10 TB critical subset; restoring the full 200 TB estate at the same throughput would take roughly 200,000 GB / 1 GB/s ≈ 55.6 hours, which is why the RTO commitment applies per-dataset-tier, not as a promise to restore everything in 4 hours.

Cataloging. Maintain a backup catalog (an index of what was backed up, when, where, and with what checksum) as a small, independently replicated service, not embedded inside the 200 TB of bulk data itself. If the catalog only exists in the data center that just failed, nobody can find anything to restore even though the data may be intact elsewhere; replicate the catalog to every data center that could need to drive a restore.

Verification. Automated, scheduled restore tests: sample a set of volumes weekly, restore them into an isolated environment, and compare checksums against what the catalog recorded at backup time. Separately, run periodic bit-rot scans against the backup storage itself, since 200 TB sitting mostly cold for months can silently degrade without ever being touched by a restore. Neither of these is optional at this scale: an untested backup is a hypothesis, not a backup.

Restore runbooks. Document, per tier: who declares an incident and authorizes a restore, the priority order (highest-priority datasets first, since restore bandwidth is a shared, finite resource across a 200 TB estate and can't restore everything in parallel at full speed), the exact automated restore procedure (scripted, not ad hoc commands typed under pressure), and the rollback point if the restore itself needs to be aborted.

How SREs operate and test it. Track backup success rate, backup duration, and restore-verification pass rate as monitored SLOs (Service Level Objectives), with alerts on any missed window. Run quarterly full DR game days that actually fail a critical dataset over to the second data center and time every runbook phase against the 4-hour and 1-hour targets, rather than trusting the design math alone.

Direct answer

When a petabyte-scale WAN backup keeps blowing past its window, the fix that matters most is almost never "add more bandwidth" or "compress harder": it's turning the backup from a repeated full transfer into a deduplicated, incremental one, because that changes the amount of data crossing the WAN by an order of magnitude, while every other lever (compression, parallel streams, scheduling) only shaves a constant factor off whatever volume you chose to send.

Structured elaboration

Global/source-side deduplication. Deduplicating at the source, before data crosses the WAN, means only genuinely unique bytes are ever transmitted; blocks identical to something already backed up (common across similar files, repeated database pages, near-duplicate snapshots) are represented by a small reference instead of being resent. This is the single highest-leverage change for a recurring, blown-window backup because it attacks the actual bottleneck (bytes over a bandwidth-limited link), not the transfer mechanism.

Block-level incremental (delta) transfer. Once a full baseline exists, every subsequent run should only ship the blocks that changed since the last run, not a fresh full copy of everything. Combined with dedup, this is what turns a petabyte-scale nightly job into a job proportional to that day's actual churn rate rather than the total dataset size.

Physical/offline seeding for the initial baseline. The very first full transfer of a petabyte-scale dataset over a bandwidth-limited WAN link is often infeasible to complete in any reasonable window at all (the worked example below shows why); shipping the initial baseline via a physical storage transfer to the target site, then switching to incremental WAN transfer for everything after that, avoids trying to solve a one-time problem with a permanent-capacity fix.

Compression. Standard compression typically yields a real but modest reduction (often in the range of two to three times smaller, heavily dependent on how compressible the data already is), which helps but does not by itself close a large window overrun the way volume reduction from dedup and incrementals does.

Bandwidth shaping, scheduling, and parallelism. Running more concurrent streams and scheduling the job to avoid competing with other WAN traffic can use the available link more fully, but these levers are capped by the physical link's total capacity; they help you use 100% of what you have, they cannot manufacture bandwidth you don't have the way reducing the bytes you need to send does.

Which change matters most. For a backup that has been consistently missing its window (not a one-off anomaly), the dominant fix is deduplication plus incremental transfer, because it reduces the transferred volume itself; compression, parallelism, and scheduling are worth doing too, but they're secondary because they optimize how efficiently you move whatever volume you decided to send, not how much volume that is.

Worked example

Take a 2 PB dataset (using decimal units throughout for a consistent basis: 1 GB = 10^9 bytes, 1 PB = 1,000,000 GB) over a 10 Gbps WAN link. Converting the link speed to the same byte basis: 10 Gbps = 10 x 10^9 bits per second / 8 = 1.25 GB/s. A full transfer of the entire 2,000,000 GB dataset at that rate takes 2,000,000 / 1.25 = 1,600,000 seconds, about 444 hours, about 18.5 days: nowhere close to any nightly or even weekly window, and this is before accounting for the fact you'd have to redo this every single run if you kept sending full copies. Now suppose the dataset has a 1% daily change rate. An incremental, deduplicated run only needs to send the changed 1% = 20,000 GB. At the same 1.25 GB/s, that takes 20,000 / 1.25 = 16,000 seconds, about 4.4 hours: comfortably inside a typical overnight window. The volume reduction from moving to incrementals (from 2,000,000 GB down to 20,000 GB, a 100x reduction in what needs to cross the WAN) dwarfs what compression (roughly 2-3x) or added parallel streams (bounded by the same 1.25 GB/s link ceiling regardless of how many streams share it) could deliver on their own.

Trade-offs and pitfalls

• Deduplication and delta transfer require tracking state (what has already been sent, what changed since last run) both at the source and target; that state itself needs to be backed up and verifiable, since a corrupted change-tracking index can silently cause an incremental backup to miss data.
• Physical seeding solves the one-time baseline problem but adds operational complexity (shipping and securely handling physical media or an appliance) and a fixed delay before the target site is even up to date with the source at seed time; it is worth it once for a petabyte-scale initial load, not as an ongoing substitute for network transfer.
• Comparing "backup window duration" against "bandwidth" only makes sense once both are expressed on the same time basis and the same byte basis; a plan that quotes the link in bits per second and the dataset in bytes without converting between them (as this worked example deliberately does) is a common source of an order-of-magnitude planning error.

Direct answer

An air-gapped backup is a copy of data kept physically or logically disconnected from the production network and from anything a production compromise could reach, so an attacker (or a runaway automated process) who has taken over production still cannot delete, encrypt, or tamper with that copy, because there is no live path to it at the time of the attack. Organizations use air gaps as the outermost layer of defense-in-depth because modern ransomware routinely targets backup infrastructure directly, and an air gap is the one layer whose protection does not depend on any other layer (perimeter, endpoint, identity) having held.

Why air gaps matter for defense-in-depth

The classic guidance here is the 3-2-1-1 rule: 3 copies of data, on 2 different media types, with 1 copy offsite, and 1 copy air-gapped or otherwise immutable. Every layer before the air-gapped copy can fail: perimeter defenses can be bypassed, endpoint protection can miss a novel payload, and identity can be compromised via a phished or stolen privileged credential. An attacker who reaches domain-admin-equivalent access in a normally-connected environment can typically also reach and destroy normally-connected backups, which is exactly why real-world ransomware incidents increasingly include the backup repository as a target, not just production. The air gap's value is that it does not rely on any of those defenses continuing to hold; it removes the network path itself.

Two practical implementations

1. Logical air gap via immutable, access-isolated object storage. Use object storage with a write-once retention lock (for example, an object-lock feature in compliance mode) so that once written, an object cannot be deleted or modified until its retention period expires, not even by an account with otherwise-broad permissions. Combine this with a credential and account boundary that production's normal operating identities never hold: a separate account or tenant, deny-by-default cross-account access, and a strictly one-way replication path (production can push new backups in, nothing, including a fully compromised production credential, can delete or modify what is already there). This is "logical" rather than physical: there is no literal cable being unplugged, but access is architecturally and cryptographically isolated such that compromising production does not grant a path to alter the vaulted copy.
2. Physical or scheduled-connection air gap. Traditional tape or removable-media rotation, still common in hybrid environments, where media is physically disconnected from any network once a write completes. A cloud-native equivalent is a vaulting pattern where a secondary environment is connected only during a scheduled replication window: a nightly job establishes a one-way connection, pushes data across, then the connection and its credentials are torn down or rotated, so for the overwhelming majority of the day there is no live network path at all, minimizing the window an attacker could exploit even if they somehow reached the vault's edge.

Keeping air-gapped backups usable for restores

The practical trap with a true air gap is that "hard to reach for an attacker" often also means "hard to reach quickly during a legitimate emergency." Mitigate this with a documented, drilled break-glass retrieval process (not invented for the first time during an actual incident), retrieval credentials staged in a separate, monitored, multi-factor-gated vault distinct from production's normal credential store, and periodic test restores from the air-gapped copy on the same cadence as regular restore drills, so the air-gapped copy does not quietly become the one backup class nobody has ever actually verified is restorable.

Direct answer

Treat the two RTO/RPO numbers as two different engineering problems, not one: the 2-hour RTO for the stateless web/app tier is an infrastructure-provisioning problem solved with immutable images and IaC (no backup of the tier itself is needed, only of its definition), while the 15-minute RPO for PostgreSQL is a data-continuity problem that a nightly or even hourly backup cannot satisfy on its own and requires continuous WAL (write-ahead log) archiving or streaming replication. Recovery order follows the dependency graph: infrastructure and the database restore in parallel where possible, but the app tier cannot serve correct traffic until the database is verified, so the database restore is the critical path inside the 2-hour budget, not an independent 2-hour budget of its own.

Backup types by component

• Frontend and app servers (stateless): no data backup at all. "Backup" here means an immutable machine image or container image plus the Infrastructure-as-Code (IaC) that defines how many instances, what config, and what network wiring, stored in version control. Recovery is redeployment, not restore.
• PostgreSQL primary with replicas: a weekly or nightly full base backup, plus continuous WAL archiving (shipping WAL segments to durable storage as they are generated, not on a fixed interval) to support point-in-time recovery (PITR). Separately, keep at least one warm standby replica for fast failover, since PITR from a cold base backup is a different, slower recovery path than promoting an already-current replica.
• S3-like object storage for user uploads: object storage is typically already durable (multi-AZ replicated by the provider), so "backup" here means versioning plus cross-region replication to protect against accidental deletion, overwrite, or a regional failure, not protection against media failure.

Same-basis check on the RPO number

15 minutes RPO means: after any failure, at most 15 minutes of committed transactions may be unrecoverable. A nightly full backup alone gives an RPO of up to 24 hours, which does not meet this requirement by two orders of magnitude, so cadence has to be re-derived from the RPO target, not assumed. Continuous WAL archiving with a shipping interval well under 15 minutes (WAL segments are typically archived every few seconds to a couple of minutes under normal load) comfortably meets the target with margin; if only a fixed-interval WAL push is available rather than continuous streaming, set that interval to something like 5 minutes, not 15, because the RPO has to account for the interval plus shipping latency plus detection time, and setting the interval exactly at the SLA boundary leaves no margin for any of those.

Storage tiers

• User uploads, 30-day hot tier: keep the most recent 30 days in a standard, low-latency object storage class since that is the window most likely to be accessed or need a fast individual-object restore (accidental deletion, corruption). Interpreting the requirement as "uploads are retained in the hot tier for 30 days" (rather than "uploads are deleted after 30 days"), lifecycle older objects to a cheaper infrequent-access or archive tier rather than deleting them, unless the business has separately confirmed 30 days is the full retention period; this is stated as an assumption because the question does not specify what happens to uploads after day 30.
• Transactions, 1-year retention: recent PostgreSQL backups (say the last 1-2 weeks of base backups plus WAL) stay in fast storage for quick PITR; older backups within the 1-year window move to a cheaper archive tier where restore latency of hours is acceptable, because a 300-day-old backup is being kept for audit or compliance reasons, not for a 2-hour RTO scenario.

Recovery order and a worked RTO check

1. Provision network and load balancer baseline via IaC. This has no data dependency and can start immediately, in parallel with step 2.
2. Restore the PostgreSQL primary (the critical path): promote an existing warm standby if one survived the failure (fast, typically single-digit minutes), or restore the latest base backup and replay WAL to the most recent consistent point if the standby is also lost (slower, and must be timed).
3. Verify database integrity (checksums, row counts, application-level sanity queries) before pointing anything at it.
4. Deploy app servers from the pinned image/IaC pointed at the verified database; this is typically fast (minutes) since it is just infrastructure provisioning with no data to move.
5. Deploy frontend web servers, same mechanism.
6. Warm any caches, then cut traffic over.

Worked check on whether cold restore fits the 2-hour budget: suppose the database is 500 GB and cold restore-plus-replay throughput from the backup store is, as an illustrative ESTIMATE, 500 MB per minute. $500000 / 500 = 1000$ minutes, about 16.7 hours, which massively breaches the 2-hour RTO even before the stateless tiers are provisioned. This is the concrete reason a cold base-backup restore should not be the primary recovery path for the app tier's 2-hour RTO: keep a warm standby that can be promoted in minutes as the primary recovery mechanism, and treat cold PITR restore as the fallback for a failure mode a standby cannot cover, such as logical corruption replicated to the standby, where the RTO target may need to be renegotiated with the business since a large cold restore genuinely cannot fit inside 2 hours at typical restore throughput.

Where the numbers do not directly compare

The RTO (2 hours, for web/app) and RPO (15 minutes, for the database) are not the same measurement and should not be added or averaged: RTO is "how long until service is back," RPO is "how much data could be lost." A design can meet both independently (standby promotion gives a low RTO; continuous WAL archiving gives a low RPO) but meeting one does not imply anything about the other, and a plan that only discusses one while citing both numbers has not actually answered the question.

Direct answer

At 2,000 VMs, sequential restore is not a plan, it is a multi-week outage, so the answer has to be a tiering scheme executed in parallel within each tier, gated by containment and by trust in the backup itself (the scenario states some backups were also encrypted, which means the backup infrastructure was reachable by the attacker and every "clean" backup must be verified before it is trusted, not assumed clean because it predates the encryption event).

Immediate containment

• Isolate affected network segments to stop lateral spread: disable inter-host SMB/RDP where feasible, segment VLANs, and pull the affected segment's route to the rest of the network rather than shutting everything down blindly.
• Rotate all privileged credentials immediately, especially anything with domain admin or backup-admin rights, since ransomware operators that reach 2,000 VMs typically did so via a compromised privileged account, not purely via a worm.
• Power off (do not destroy) actively encrypting hosts to stop further damage while preserving forensic evidence on disk; a live host still encrypting is actively destroying data every additional minute it stays up.
• Because "some backups" were encrypted, immediately isolate and lock down the backup infrastructure itself (separate credentials, separate network path) and treat every backup repository as suspect until integrity-checked, not just the ones known to be hit.
• Preserve evidence: snapshot volatile state and logs from a representative sample of affected hosts before any restore activity overwrites it, since the same incident will need a root-cause and likely a breach investigation.

Prioritization criteria (why AD and DNS come first)

Everything else depends on identity and name resolution, so restoring an application server before its supporting Active Directory (AD) domain controllers and DNS are trustworthy just means restoring a system that cannot authenticate or be found. Tiering:

1. Tier 0, trust root: AD domain controllers, DNS, DHCP, time sync (NTP), and the backup, EDR (endpoint detection and response, the agent that watches hosts for malicious behavior), and SIEM (security information and event management, the centralized log and alert aggregator) infrastructure itself. These are restored from a verified-clean backup taken before the earliest known indicator of compromise, not simply "yesterday's" backup, because ransomware frequently dwells silently for days to weeks before triggering encryption, and restoring from a backup taken during that dwell time reintroduces the same backdoor.
2. Tier 1, core infrastructure: hypervisor management plane, network devices/firewalls, security tooling.
3. Tier 2, business-critical applications: ranked by a pre-existing business impact analysis (revenue-generating systems, safety-critical systems, systems under regulatory SLA), not restored ad hoc by whoever asks loudest.
4. Tier 3, everything else: restored by department priority and dependency order once the tiers above are stable.

Recovery steps, per VM

For each VM: identify a backup generation confirmed to predate the earliest compromise indicator; restore it into an isolated recovery network with no route to production; run anti-malware/EDR scanning and file-integrity checks against it; identify and close the specific vulnerability believed to have allowed the initial compromise (unpatched service, exposed credential, phishing-delivered payload) before reconnecting; rotate any credentials that lived on that VM; only then move it onto the production network.

Validation

Before declaring a restored VM production-ready: automated boot and health check; malware/EDR scan clean; checksum or hash comparison of critical files against known-good; application-level smoke tests for anything customer-facing; and an observation window (hours, not minutes) watching for re-encryption or beaconing behavior before the VM is trusted unattended, since a restored-but-still-backdoored host will often stay quiet until it reconnects to its command-and-control infrastructure.

Stakeholder communication

A single incident commander owns messaging so technical recovery and business communication do not conflict. A pre-agreed cadence (for example every 30-60 minutes during the active phase) goes to executives, legal, and PR. Because most modern ransomware is double-extortion (data exfiltrated, not only encrypted), legal must assess breach-notification obligations early rather than waiting for full recovery, and customer-facing status updates should be honest about scope without disclosing details that would help the attacker or violate the ongoing investigation. Recovery progress should be reported by tier ("Tier 0 identity restored and verified, Tier 1 in progress") since that is a metric stakeholders can track and it maps to what is actually unblocking further recovery.