Backup and Disaster Recovery Questions

Direct answer

At 200 TB, two things break the naive reading of the requirements and have to be designed around explicitly. First, "weekly full backup" cannot mean literally re-reading and re-copying all 200 TB every week: at a realistic sustained backup throughput of 500 MB/s, copying 200 TB takes roughly 200,000,000 MB / 500 MB/s ≈ 400,000 seconds ≈ 111 hours, well over four days, so a literal weekly full would never finish before the next one starts. The fix is snapshot-based, copy-on-write "synthetic full" backups: a full recovery point that is constructed from an incremental chain without re-reading the whole dataset. Second, RPO < 1 hour (Recovery Point Objective: the maximum acceptable data loss, measured as time) is stated only for the highest-priority subset of data, not the whole 200 TB; daily incrementals alone give an RPO of up to 24 hours, so that highest-priority subset needs a separate, more frequent capture mechanism layered on top of the daily/weekly baseline, not a tighter version of the same schedule applied everywhere.

Structured elaboration

Architecture: snapshot vs block-level copy vs agent, and why a hybrid.

• Storage-level snapshots (copy-on-write snapshots at the SAN, Storage Area Network, a dedicated network of shared block-storage devices, array, or cloud block-storage layer) are fast to create, capture only changed blocks, and are the right mechanism for the frequent, high-priority tier: they don't require reading the whole volume, so an hourly snapshot on a 200 TB estate is cheap regardless of total size. They are typically crash-consistent by default (consistent with an ungraceful power-off) and need an application-aware quiesce step (flushing writes, briefly freezing the filesystem) to be application-consistent for databases.
• Block-level backup/replication (deduplicated, changed-block backup software, or asynchronous block replication to the second data center) is what actually gets data across data centers for disaster recovery, since a local snapshot alone doesn't survive losing the data center it lives in. This is the mechanism that should carry the highest-priority subset's near-real-time copy to the DR site.
• Agent-based, file-level backup (an agent running inside each host or VM that understands the application, e.g. taking an application-consistent dump via a pre-freeze hook) is the most flexible but the slowest to scan at this scale, and shouldn't be the primary mechanism for the bulk of 200 TB. Reserve it for stateful applications (databases, message queues) that need an application-consistent capture the block layer can't provide on its own, layered on top of block-level backup for the rest.
• Recommended hybrid: block-level, copy-on-write snapshots as the default for all volumes (daily, feeding synthetic weekly fulls); asynchronous block-level replication to the second data center for the highest-priority subset, running frequently enough (e.g. every 15 to 30 minutes) to comfortably clear the 1-hour RPO with margin; agent-based, application-consistent backups only for databases and similar stateful services that need a coordinated quiesce, at the same cadence as their tier.

Worked restore-time example (basis: a single critical dataset's volume, not the full 200 TB). Suppose the highest-priority tier is a 10 TB subset. At an aggregate parallel restore throughput of 1 GB/s (achievable by restoring many volumes or shards concurrently rather than serially), restoring 10 TB takes 10,000 GB / 1 GB/s = 10,000 s ≈ 2.8 hours, comfortably inside the 4-hour RTO (Recovery Time Objective: the maximum acceptable downtime) for that dataset. This number is scoped to the 10 TB critical subset; restoring the full 200 TB estate at the same throughput would take roughly 200,000 GB / 1 GB/s ≈ 55.6 hours, which is why the RTO commitment applies per-dataset-tier, not as a promise to restore everything in 4 hours.

Cataloging. Maintain a backup catalog (an index of what was backed up, when, where, and with what checksum) as a small, independently replicated service, not embedded inside the 200 TB of bulk data itself. If the catalog only exists in the data center that just failed, nobody can find anything to restore even though the data may be intact elsewhere; replicate the catalog to every data center that could need to drive a restore.

Verification. Automated, scheduled restore tests: sample a set of volumes weekly, restore them into an isolated environment, and compare checksums against what the catalog recorded at backup time. Separately, run periodic bit-rot scans against the backup storage itself, since 200 TB sitting mostly cold for months can silently degrade without ever being touched by a restore. Neither of these is optional at this scale: an untested backup is a hypothesis, not a backup.

Restore runbooks. Document, per tier: who declares an incident and authorizes a restore, the priority order (highest-priority datasets first, since restore bandwidth is a shared, finite resource across a 200 TB estate and can't restore everything in parallel at full speed), the exact automated restore procedure (scripted, not ad hoc commands typed under pressure), and the rollback point if the restore itself needs to be aborted.

How SREs operate and test it. Track backup success rate, backup duration, and restore-verification pass rate as monitored SLOs (Service Level Objectives), with alerts on any missed window. Run quarterly full DR game days that actually fail a critical dataset over to the second data center and time every runbook phase against the 4-hour and 1-hour targets, rather than trusting the design math alone.

Direct answer

Treat the two RTO/RPO numbers as two different engineering problems, not one: the 2-hour RTO for the stateless web/app tier is an infrastructure-provisioning problem solved with immutable images and IaC (no backup of the tier itself is needed, only of its definition), while the 15-minute RPO for PostgreSQL is a data-continuity problem that a nightly or even hourly backup cannot satisfy on its own and requires continuous WAL (write-ahead log) archiving or streaming replication. Recovery order follows the dependency graph: infrastructure and the database restore in parallel where possible, but the app tier cannot serve correct traffic until the database is verified, so the database restore is the critical path inside the 2-hour budget, not an independent 2-hour budget of its own.

Backup types by component

• Frontend and app servers (stateless): no data backup at all. "Backup" here means an immutable machine image or container image plus the Infrastructure-as-Code (IaC) that defines how many instances, what config, and what network wiring, stored in version control. Recovery is redeployment, not restore.
• PostgreSQL primary with replicas: a weekly or nightly full base backup, plus continuous WAL archiving (shipping WAL segments to durable storage as they are generated, not on a fixed interval) to support point-in-time recovery (PITR). Separately, keep at least one warm standby replica for fast failover, since PITR from a cold base backup is a different, slower recovery path than promoting an already-current replica.
• S3-like object storage for user uploads: object storage is typically already durable (multi-AZ replicated by the provider), so "backup" here means versioning plus cross-region replication to protect against accidental deletion, overwrite, or a regional failure, not protection against media failure.

Same-basis check on the RPO number

15 minutes RPO means: after any failure, at most 15 minutes of committed transactions may be unrecoverable. A nightly full backup alone gives an RPO of up to 24 hours, which does not meet this requirement by two orders of magnitude, so cadence has to be re-derived from the RPO target, not assumed. Continuous WAL archiving with a shipping interval well under 15 minutes (WAL segments are typically archived every few seconds to a couple of minutes under normal load) comfortably meets the target with margin; if only a fixed-interval WAL push is available rather than continuous streaming, set that interval to something like 5 minutes, not 15, because the RPO has to account for the interval plus shipping latency plus detection time, and setting the interval exactly at the SLA boundary leaves no margin for any of those.

Storage tiers

• User uploads, 30-day hot tier: keep the most recent 30 days in a standard, low-latency object storage class since that is the window most likely to be accessed or need a fast individual-object restore (accidental deletion, corruption). Interpreting the requirement as "uploads are retained in the hot tier for 30 days" (rather than "uploads are deleted after 30 days"), lifecycle older objects to a cheaper infrequent-access or archive tier rather than deleting them, unless the business has separately confirmed 30 days is the full retention period; this is stated as an assumption because the question does not specify what happens to uploads after day 30.
• Transactions, 1-year retention: recent PostgreSQL backups (say the last 1-2 weeks of base backups plus WAL) stay in fast storage for quick PITR; older backups within the 1-year window move to a cheaper archive tier where restore latency of hours is acceptable, because a 300-day-old backup is being kept for audit or compliance reasons, not for a 2-hour RTO scenario.

Recovery order and a worked RTO check

1. Provision network and load balancer baseline via IaC. This has no data dependency and can start immediately, in parallel with step 2.
2. Restore the PostgreSQL primary (the critical path): promote an existing warm standby if one survived the failure (fast, typically single-digit minutes), or restore the latest base backup and replay WAL to the most recent consistent point if the standby is also lost (slower, and must be timed).
3. Verify database integrity (checksums, row counts, application-level sanity queries) before pointing anything at it.
4. Deploy app servers from the pinned image/IaC pointed at the verified database; this is typically fast (minutes) since it is just infrastructure provisioning with no data to move.
5. Deploy frontend web servers, same mechanism.
6. Warm any caches, then cut traffic over.

Worked check on whether cold restore fits the 2-hour budget: suppose the database is 500 GB and cold restore-plus-replay throughput from the backup store is, as an illustrative ESTIMATE, 500 MB per minute. $500000 / 500 = 1000$ minutes, about 16.7 hours, which massively breaches the 2-hour RTO even before the stateless tiers are provisioned. This is the concrete reason a cold base-backup restore should not be the primary recovery path for the app tier's 2-hour RTO: keep a warm standby that can be promoted in minutes as the primary recovery mechanism, and treat cold PITR restore as the fallback for a failure mode a standby cannot cover, such as logical corruption replicated to the standby, where the RTO target may need to be renegotiated with the business since a large cold restore genuinely cannot fit inside 2 hours at typical restore throughput.

Where the numbers do not directly compare

The RTO (2 hours, for web/app) and RPO (15 minutes, for the database) are not the same measurement and should not be added or averaged: RTO is "how long until service is back," RPO is "how much data could be lost." A design can meet both independently (standby promotion gives a low RTO; continuous WAL archiving gives a low RPO) but meeting one does not imply anything about the other, and a plan that only discusses one while citing both numbers has not actually answered the question.

Direct answer

RPO (Recovery Point Objective: the maximum data-loss window) under 1 minute is tight enough to change the design decision compared to a looser target: at this ceiling, asynchronous replication's typical seconds-level lag is often fine in practice, but it isn't a guaranteed bound, replication lag can spike well past a minute under load or a network hiccup with nothing stopping it. Meeting RPO < 1 minute as an actual guarantee, not just a usual outcome, points toward synchronous or quorum-based replication (the primary waits for acknowledgment from the standby, or from a quorum of standbys, before committing), accepting the added write latency that comes with it as the deliberate cost of the guarantee.

Structured elaboration

Replication method. Synchronous replication (e.g. Postgres synchronous_commit=on or remote_apply) gives RPO effectively zero for infrastructure failures, since no commit is acknowledged to the client until the standby has it, but adds the cross-region round-trip time (commonly tens to over a hundred milliseconds depending on the region pair) to every write, which can become a real throughput constraint under high transaction rates. Quorum-based replication (a write is acknowledged once a quorum, e.g. 2 of 3 regional replicas, has it, not all of them) gives the same strong guarantee while tolerating one slow or unreachable replica without stalling the primary. A well-monitored asynchronous setup, with alerting on lag well inside the 1-minute budget, is a real option only if the business explicitly accepts "usually under a minute, not contractually guaranteed under adverse conditions"; that trade-off should be surfaced explicitly rather than silently assumed away by picking async and hoping.

Failover procedure and RTO budget. Detection (confirmed failure, not a single blip): roughly 2 minutes. Promotion: roughly 3 minutes, faster than a looser-RPO design because a synchronous or quorum replica is, by construction, already caught up and doesn't need catch-up time before it's safe to promote. Traffic cutover: roughly 5 minutes, requiring a short DNS TTL (time-to-live, how long clients cache a DNS record before re-checking it) set in advance (e.g. 30 to 60 seconds) plus application-level reconnect/retry behavior that doesn't require a manual restart. Validation smoke test before declaring recovery: roughly 3 minutes. Summed: roughly 13 minutes, leaving about 2 minutes of margin under the 15-minute RTO (Recovery Time Objective: the maximum downtime) target, all figures illustrative and meant to be re-timed in a real drill, not asserted as guaranteed.

Data consistency considerations. With synchronous or quorum replication, the promoted standby is guaranteed consistent up to the last acknowledged commit, so failover requires no reconciliation, the biggest consistency advantage over asynchronous replication. The real risk with any replication mode is the old primary coming back online after failover with writes the new primary never received (a split-brain divergence): it has to be explicitly fenced (its ability to accept writes revoked) the moment failover is declared, and any writes it accepted after the split should be treated as lost or requiring manual reconciliation, not silently merged back in.

Testing DR with minimal production impact. Promote a replica into an isolated network segment for drills, not the live traffic path, so testing the mechanics of failover never touches real user traffic. Validate the promoted instance with synthetic or replayed read-only traffic, or a canary write workload against a namespaced test schema, rather than pointing real users at it. To test the cutover mechanism itself (DNS or routing changes) without customer impact, use a small canary percentage of traffic, or hit the DR region's endpoint directly with an internal synthetic client, bypassing the global router that serves real users, so the target environment is validated without touching what actually routes production. Time every phase of each drill against the 15-minute budget and update the runbook if any phase trends over its allotted share.

Direct answer

Restoring a Kubernetes cluster is really two coordinated restores that have to be sequenced correctly: the cluster's control-plane state (etcd) has to come back first since nothing else can be reconciled without it, while the actual application data on persistent volumes is restored at the storage layer and then re-linked to the cluster objects that expect it, and secrets, storage classes, and bootstrap order each carry their own specific failure modes that a generic "restore everything" plan misses.

Structured elaboration

Restoring etcd. etcd is the key-value store holding the entire desired-state of the cluster (every Deployment, ConfigMap, Secret, and other object definition); the standard recovery path is restoring from an etcd snapshot onto fresh data directories for every etcd member, which always forms a new etcd cluster rather than rejoining the old one. All members must be restored from the exact same snapshot; mixing snapshots taken at different points across members produces an internally inconsistent cluster state. Once etcd is restored, point the API server(s) at it and confirm the API server comes up healthy before proceeding.

Persistent volumes for statefulsets. Restoring the volumes backing StatefulSet pods happens at the storage layer, independently of etcd, since it's usually a completely different system (a cloud block-storage service, an in-cluster storage system) with its own backup and restore mechanism. This independence is exactly what makes ordering tricky: if etcd is restored to a point that doesn't match what actually exists at the storage layer after volumes are restored, especially if restored volumes come back under new identifiers, Kubernetes' record of which PersistentVolume (PV, the actual storage resource) backs which PersistentVolumeClaim (PVC, a pod's request for storage that gets bound to a PV) can point at a volume ID that either doesn't exist anymore or, worse, now refers to different data than the cluster's restored state expects. The restore plan needs an explicit remapping step: after restoring volumes at the storage layer, update or recreate the PersistentVolume objects so their volume handles correctly point at the restored volumes, rather than assuming the old references still resolve correctly.

Re-creating cluster resources. After etcd and the API server are healthy, worker node kubelets (the per-node agent that runs and manages pods on that node) reconnect and re-register, and the scheduler and controllers begin reconciling the restored desired state against actual running pods. Some things aren't fully captured by an etcd snapshot alone and may need separate recreation alongside it: infrastructure-managed resources like external DNS records or cloud load balancer bindings that a service or ingress object depends on, which are often owned by infrastructure-as-code outside the cluster and need their own restore or reconciliation step run in parallel.

Secrets: KMS and rotation. Kubernetes Secrets are commonly encrypted at rest in etcd using an encryption-at-rest configuration backed by a key management service (KMS). The restored etcd snapshot contains ciphertext, so the exact key (or key version) that was active at backup time must still exist and be accessible at restore time, or those secrets are permanently unreadable regardless of how successful the rest of the restore was. This is a real, easy-to-miss gap: key rotation policy has to explicitly retain old key versions for at least the full backup retention window, or re-encrypt all secrets under a new key before an old one is destroyed, and it's worth periodically testing a restore against an intentionally older backup specifically to catch a rotated-and-destroyed key before it's discovered during a real incident.

StorageClass differences across providers. If the restore target is a different cluster, or the same cluster on a different cloud provider or storage backend than where the backup was taken, StorageClass names and their underlying provisioners often don't match; a PersistentVolumeClaim bound via a specific provider's StorageClass on the source may reference a provisioner that doesn't exist, or exists with different parameters, on the target. Restore tooling needs to remap StorageClass names to their equivalents on the target and verify performance-tier parity explicitly (a "fast" tier on the source silently landing on a slower equivalent on the target is not something a "restore succeeded" status check will catch).

Bootstrap steps. A reasonable sequence: restore etcd across all members from the same snapshot, start the API server(s) against the restored etcd and confirm cluster health, restore or remap the persistent volumes at the storage layer and update PersistentVolume objects to match, verify KMS access for encrypted secrets before relying on anything that depends on them, allow kubelets to reconnect, bring up cluster-critical system components first (networking/CNI, meaning the Container Network Interface, the plugin that provides pod networking, plus DNS, ingress controller) since application pods depend on those being functional, then allow StatefulSet pods to bind to the restored, remapped volumes, and finally run an application-level smoke test before declaring the cluster recovered rather than trusting Kubernetes-level "pod is Running" status alone.

Worked example

A team restores a cluster after a regional outage destroyed the original. They restore all 5 etcd members from the same hourly snapshot and bring the API server up first, confirming kubectl get nodes and core objects look correct before touching anything else. Storage-layer volume restore runs in parallel: block volumes come back under new provider-assigned IDs, so the team runs a remapping script that updates each PersistentVolume object's volume handle to the new ID before letting any StatefulSet pod schedule. KMS access is verified next: the backup was taken before a key rotation two weeks ago, and because the team's rotation policy explicitly retains prior key versions for 90 days, the old key version is still available and Secrets decrypt correctly, avoiding what would otherwise have been a silent, hard-to-diagnose failure only surfacing when an application tried to read a secret. CNI and CoreDNS come up first once kubelets reconnect; only after those report healthy does the team allow StatefulSet pods to schedule against the remapped volumes, followed by an application-level health check before declaring the restore complete.

Trade-offs and pitfalls

• Restoring etcd from a snapshot taken at a slightly different time than the storage-layer volume backups is a common, subtle failure: the cluster's record of what should exist and the actual data on restored volumes can silently disagree, which is why volume remapping needs to be an explicit, verified step, not an assumption.
• A restore drill that always uses the most recently rotated encryption key will never catch a retention gap in older key versions; testing restores against a deliberately older backup point is the only way to catch that specific failure mode before it happens for real.
• Bringing StatefulSet pods online before cluster-critical networking and DNS components are healthy can cause those pods to crash-loop or come up in a degraded state that looks like a data problem but is actually a sequencing problem, wasting time on the wrong root cause during an already-stressful recovery.

Direct answer

The number that determines the design here is RPO = 1 hour against RTO = 30 minutes (RPO: Recovery Point Objective, maximum acceptable data loss measured as time; RTO: Recovery Time Objective, maximum acceptable downtime), and the key calculation is this: a 30-minute RTO rules out "restore from a base backup and replay the WAL (write-ahead log, Postgres's append-only record of every change) since then" as the primary failover mechanism, because replay time scales with how much WAL has accumulated, and that can easily exceed 30 minutes. If the cluster generates roughly 10 GB/hour of WAL and a base backup is taken every 6 hours, the worst-case backlog to replay is about 60 GB; even at a generous replay throughput of 500 MB/minute, that's 60,000 MB / 500 MB/min = 120 minutes, four times the RTO budget. So the failover path has to be promoting an already-caught-up standby, not restoring from backup, and the base backup plus archived WAL exist for a different purpose: bounding retention, enabling point-in-time recovery to an arbitrary past moment, and bootstrapping new standbys, not for meeting the 30-minute RTO during an incident.

Structured elaboration

Backup cadence. A daily pg_basebackup (a full physical copy of the cluster) as the retention/PITR anchor, plus continuous WAL streaming to both the standby (for replication) and to archival object storage (for point-in-time recovery and as the bootstrap source for building new standbys later). The daily cadence is sized for retention and PITR, not for the failover RTO, per the calculation above.

WAL shipping. WAL segments stream continuously to the standby via Postgres streaming replication, and separately archive to object storage (commonly via pgBackRest or wal-g) on a short interval, e.g. every few seconds to a minute, so PITR granularity stays well inside the 1-hour RPO with margin.

Sync vs async replication. Given RPO = 1 hour, asynchronous replication is more than sufficient: typical cross-region replication lag under normal conditions is seconds, occasionally spiking to low minutes under load, both far inside the 1-hour budget. Synchronous replication (synchronous_commit with synchronous_standby_names) would guarantee near-zero RPO, but adds the cross-region round-trip latency to every commit and risks stalling the primary if the standby is unreachable, a real cost that buys nothing extra against a 1-hour target. Use async as the default, and set a lag-based alert well inside the budget, for example paging at 15 minutes of observed lag (a quarter of the 1-hour budget), so operators have real time to react before the RPO commitment is actually at risk.

Failover orchestration. Automated health checks require multiple consecutive failures over a defined window before triggering, to avoid a single network blip causing an unnecessary failover; on confirmed failure, promote the standby (pg_ctl promote or the platform's managed failover call), repoint the client-facing endpoint, and fence the old primary (revoke its ability to accept writes) once the standby is confirmed primary, preventing a split-brain if the old primary later becomes reachable again.

Data validation. Before declaring the incident resolved, confirm the replication lag at the moment of promotion (this tells you the actual data-loss window, which should be checked against the 1-hour RPO, not assumed), run row-count or checksum spot checks on critical tables against a known-good pre-incident baseline if one exists, and validate application-level invariants, e.g. that the latest committed transaction IDs are sane and sequential.

Rehearsal. Run a quarterly game day that actually promotes the standby into an isolated environment, timing every phase (detection, promotion, cutover, validation) against the 30-minute target end to end, since summing individually-plausible phase estimates doesn't guarantee the total holds. Separately and independently, rehearse the cold path too, restoring the daily base backup plus its archived WAL into a fresh instance, even though it isn't the primary failover mechanism, because an archive that's never been restored is unverified, and if it's silently broken you won't find out until you actually need PITR to an arbitrary past point.