Backup and Disaster Recovery Questions
Keeping data durable and recoverable when systems fail: backup design (full, incremental, differential, and snapshot strategies; point-in-time recovery), backup verification and restore testing, retention and archival policy (including compliance retention and legal holds), encryption and key management for backups, and disaster-recovery planning measured against recovery-time and recovery-point objectives (RTO/RPO). Tests whether a candidate can design a backup strategy that actually restores, choose the right retention tier for a business's downtime and data-loss tolerance, and operate backup systems safely under failure, compliance, and ransomware threats. Distinct from code-level fault-tolerance patterns (circuit breakers, retries, bulkheads) and multi-region failover architecture, which belong to high-availability-and-disaster-recovery.
Architect an enterprise backup and recovery solution for a hybrid environment with 10,000 VMs and 50 PB of data across on-prem NAS and public cloud object stores. Requirements: tiered RTO/RPO for multiple service tiers, global deduplication, encryption, immutable retention for 7 years, GDPR controls, and an automated quarterly restore test plan. Provide a high-level design, key components, data flow, and testing/automation strategy.
Sample Answer
Direct answer
A 10,000-VM, 50 PB hybrid backup and recovery solution has to be built around per-tier RTO/RPO commitments rather than one blanket target, a global deduplication fabric spanning both on-prem and cloud sources to make the storage cost tractable at that scale, encryption and immutability enforced together for the regulated long-term tier, and an explicit, engineered answer to the real tension between GDPR's right-to-erasure and immutable, deduplicated, shared retention, none of which resolve themselves by default.
Structured elaboration
Tiered RTO/RPO for multiple service tiers. Define distinct service tiers rather than one target for the whole 50 PB estate: for example a critical tier with an RTO around 1 hour and RPO around 15 minutes, a standard tier with an RTO of several hours and RPO around an hour, and an archival or cold tier with an RTO measured in a day or more and RPO around a day, each mapped to a different backup frequency, replication strategy, and storage class. A single blanket target across 10,000 heterogeneous VMs would either be wildly over-engineered and expensive for low-priority systems or dangerously insufficient for the genuinely critical ones.
Global deduplication. A shared deduplication fabric, using content-defined chunking and a global hash index, spans backups sourced from both on-prem NAS and cloud object stores, so identical data appearing across different VMs and sources (shared base operating-system images, common libraries, duplicated datasets) is stored once regardless of which side of the hybrid environment it originated from. At this scale, deduplication ratios in VM-heavy estates are commonly reported in the range of roughly 5 to 20 times, though the real ratio depends heavily on the specific workload mix and should be measured against this environment's actual data, not assumed from a general figure.
Encryption. Encrypt data both in transit, for any cross-site and cross-cloud replication, and at rest, across the chunk store and the metadata store, with a key-management system that rotates keys on a defined schedule while explicitly retaining historical key versions for the full length of the retention window; a rotated-and-destroyed key makes every backup encrypted under it permanently unreadable, a real, avoidable failure mode rather than a theoretical one.
Immutable retention for 7 years. For the regulatory long-term retention tier, back the retained copies with a genuine immutability mechanism, write-once/object-lock style enforcement, so that no operator mistake or compromised credential can delete or shorten that retention before the 7-year period elapses. This immutability guarantee has to be honored by the deduplication and garbage-collection logic too. In a dedup system, unique data is stored once as chunks; each backup's manifest lists which chunks it needs; and a reference count per chunk tracks how many manifests still point to it. Concretely: a chunk still referenced by an immutable-tier manifest must never be reclaimed (deleted by the cleanup process once nothing needs it), regardless of what reference counts on other, non-immutable tiers might otherwise suggest.
GDPR controls. GDPR's right to erasure sits in direct, structural tension with both 7-year immutable retention and with deduplication (a chunk shared across many tenants or data subjects can't simply be deleted to satisfy one subject's erasure request without affecting everyone else who shares it). The practical resolution most organizations use is twofold: encrypt each tenant's or data subject's data under its own distinct key, so that destroying the key ("crypto-shredding") renders that data permanently unreadable without needing to physically delete shared, deduplicated bytes that other tenants still legitimately reference; and maintain a clear, documented policy establishing backup retention as a defined, time-boxed, legitimate purpose, since most data-protection frameworks including GDPR recognize backup retention as an accepted exception, but only when the organization has actually documented and can point to that policy, not left it unaddressed.
Restore orchestration, testing, and automation strategy. An automated restore-testing pipeline, the same underlying shape as a continuous backup-verification pipeline scaled to enterprise scope, runs a scheduled sample every quarter across the fleet: full coverage (100%) of the critical tier every quarter given its small size and outsized importance, and a statistically representative, rotating sample of the standard and archival tiers, structured so that full coverage of the entire fleet completes within a bounded, multi-quarter cycle rather than never actually happening for the lower tiers. Each test performs a real restore into an isolated environment, validates at the application level, and reports results automatically to a dashboard, opening a ticket on any failure rather than requiring a human to notice.
High-level design, key components, and data flow. Putting the pieces above together into one pipeline: lightweight ingestion agents run on each VM and against each NAS source, performing source-side content-defined chunking and deduplication before anything crosses the network; chunks flow over encrypted transport into a tiered chunk store spanning hot, warm, cold, and archive classes across both on-prem and cloud capacity; a separate, globally-sharded metadata and index store tracks chunk hashes, reference counts, and per-backup manifests, and is what every ingest and restore operation actually queries; a lifecycle manager sits above both stores enforcing tiering transitions, WORM immutability on the regulated tier, and per-tenant key scoping for GDPR; a restore orchestrator handles the time-budgeted, parallel-fan-out restore sequence for any given service's RTO tier; and the automated test harness described above continuously exercises that same restore orchestrator against real, isolated targets rather than sitting off to the side as a separate, disconnected system.
Worked example
For the critical tier, comprising roughly 500 of the 10,000 VMs running the organization's core transactional services, the quarterly test plan restores all 500 into an isolated environment each quarter and validates application-level health on each. For the remaining 9,500 standard and archival VMs, the plan rotates a sample of roughly 800 per quarter, weighted toward VMs that haven't been tested in the longest time, so that over a roughly 12-quarter (3-year) cycle, every VM in the fleet has been genuinely restore-tested at least once, a concrete, auditable coverage guarantee rather than an assumption that sampling alone is "probably fine." A specific tenant's erasure request under GDPR is handled by destroying that tenant's dedicated encryption key rather than attempting to physically delete their data out of shared, deduplicated storage; because their data was encrypted separately from other tenants' data specifically to make this possible, the erasure completes without touching a single byte that other tenants' backups still legitimately reference.
Trade-offs and pitfalls
- Per-tenant or per-subject encryption keys reduce the deduplication benefit somewhat, since identical raw content encrypted under different keys no longer produces identical stored bytes across tenants, a real, deliberate cost accepted specifically because it's what makes GDPR-compliant erasure practical without breaking shared, deduplicated storage for everyone else.
- A dedup ratio assumed from a general industry figure rather than measured against this specific environment's actual workload mix can make the entire storage cost model wrong by a large margin; the estimate belongs in initial planning, but the actual ratio needs to be measured early and the cost model corrected against reality.
- Applying the 1-hour critical-tier RTO commitment as if it meant "restore all 500 critical VMs simultaneously within an hour" rather than "restore any single critical VM's own footprint within an hour" is the same basis error that would appear in a smaller single-service design; a fleet-wide simultaneous-disaster scenario needs its own explicitly stated, and likely different, target.
As a senior systems administrator you must recommend whether to purchase a commercial backup platform or build an internal backup solution for a global enterprise. Create a decision framework covering total cost of ownership, SLAs, feature gaps, vendor lock-in, security and compliance, operational overhead, and a final recommendation process including pilots and evaluation criteria.
Sample Answer
Direct answer
Default to buy unless there is a specific, named reason not to: backup infrastructure is rarely a competitive differentiator, getting it wrong is close to existential risk, and a commercial vendor amortizes that risk (and the engineering behind avoiding it) across every one of its customers, not just this one enterprise. Build only when a genuine, well-documented gap exists (an unsupported source type, a compliance requirement no vendor meets, or a cost structure that is provably worse at the organization's actual scale) and the organization is willing to own that gap forever, not just through the first successful pilot.
Total cost of ownership (TCO), on the same basis
Compare like for like: total cost over a fixed multi-year horizon (3-5 years is typical for infrastructure decisions), not year-1 build cost against annual license cost, because build's year-1 cost is artificially low (not much has been built yet) while license cost is already run-rate from day one. Illustrative worked example, explicitly an ESTIMATE and not a market quote: protecting 2 PB of data.
- Buy: licensing around $0.02 per GB per month (ESTIMATE) times 2,000,000 GB gives about $40,000 per month, or $480,000 per year, plus roughly one FTE to operate at $150,000 fully loaded, for about $630,000 per year.
- Build: three senior engineers at $180,000 fully loaded each is $540,000 per year, plus infrastructure cost for the backup engine itself (ESTIMATE, around $100,000 per year), for about $640,000 per year in direct run cost, similar order of magnitude to buy.
The point of this exercise is not that the two numbers come out close (they will not always), it is that build's TCO also carries a shadow cost the run-rate comparison misses: the multi-year period before build reaches feature parity with a mature commercial product, during which the organization either operates with real feature gaps or pays for both paths at once. Any real comparison must include that ramp period, not just steady-state run cost.
SLA
A commercial vendor typically offers a contractual SLA (for example 99.9% platform availability, defined support response times) with financial penalties or credits if missed, giving external accountability. Build means the organization's own SRE team is the SLA, with no outside party to escalate to and no consequence beyond the internal cost of the outage itself.
Feature gaps
Commercial platforms usually arrive with a broad, mature feature set (deduplication, immutability/object-lock, connectors for many source systems) on day one. Build lets the organization match its exact requirements precisely but takes real time to reach parity, and some long-tail features may never be built without dedicated, ongoing investment, since they compete for the same engineering time as everything else the team owns.
Vendor lock-in
Commercial risk: proprietary backup formats can make it hard to leave without a costly migration, and pricing or support quality can change unilaterally once switching cost is high. Build risk is the mirror image: full control, but the organization now owns that control forever, including the bus-factor risk of the few engineers who understand the internals leaving.
Security and compliance
Commercial vendors commonly carry third-party attestations (SOC 2 and ISO 27001, general-purpose security-practice attestations most enterprise vendors carry, plus industry-specific ones like HIPAA for healthcare data or FedRAMP for US federal government use, where relevant) that materially reduce the organization's own audit burden, since "our vendor is independently certified" is a much shorter conversation with an auditor than "we self-attest to our own home-built system." Build means the organization must self-certify everything and carries full responsibility for the security posture of a system it wrote, with no vendor security team to lean on.
Operational overhead
Commercial (especially SaaS) delegates most patching and upgrade burden to the vendor. Build means the organization's own on-call absorbs every edge case (encryption bugs, restore corruption, scaling limits) with no vendor support line to escalate to when something genuinely strange happens at 3 a.m.
Final recommendation process: pilots and evaluation criteria
- Shortlist two to three commercial vendors plus, only if a real gap was identified above, a scoped build proof-of-concept limited to a single workload class.
- Define evaluation criteria upfront, before the pilot starts, so the process is not retrofitted to favor whichever option looked better first: RTO/RPO actually achieved in a real restore test, cost per TB at the organization's actual scale (not list price), restore success rate across several drill runs, integration effort measured in engineer-weeks, and pass/fail on the security review.
- Weight criteria by organizational priority (a compliance-heavy organization weights security/compliance higher than a small startup would).
- Time-box the pilot (roughly 6-8 weeks is typical) with a defined go/no-go gate, so the evaluation itself does not become the indefinite, unbounded project that build risk warns against.
Case study: A compliance workload requires 10-year retention and must be recoverable within 48 hours. Compare two approaches quantitatively and qualitatively: (A) keep replicated backup copies in two regions (higher ongoing storage/egress cost) vs (B) keep primary in lower-cost cold archive and perform on-demand restores to recover within 48 hours. What variables determine the better approach and how would you model costs and risk?
Sample Answer
Direct answer
Build a total-cost model over the same ten-year horizon for both approaches, using the same
dataset size on both sides, then find the break-even restore frequency at which the two costs are
equal. In a realistic model, keeping a cold archive copy and restoring on demand is dramatically
cheaper unless full-dataset restores happen roughly monthly or more often, which is far more often
than a genuine compliance archive is typically accessed. The variable that actually decides the
question is expected restore frequency, not raw storage price, though a hard regulatory
requirement for geographic redundancy can override the cost comparison entirely.
Quantitative model
Take an illustrative 8,000 GB (8 TB) compliance dataset, retained for 10 years, with a 48-hour
recovery requirement. All figures below are illustrative order-of-magnitude prices for
demonstrating the model, not live vendor pricing, and every step stays on the same basis: GB,
GB-month, and dollars per year, converted consistently.
Approach A: replicated copies in two regions, on a tier that supports near-immediate restore,
at an illustrative $0.01 per GB-month per region.
- Per region: 8,000 GB times $0.01 per GB-month is $80 per month.
- Two regions combined: $160 per month, or $1,920 per year, or $19,200 over 10 years.
- Plus a one-time cross-region seed transfer of roughly 8,000 GB at an illustrative $0.02 per GB
egress rate: about $160. - Total over 10 years: approximately $19,360.
Approach B: a single-region cold archive tier plus on-demand restores, at an illustrative
$0.001 per GB-month for storage and $0.02 per GB for a retrieval that still meets the 48-hour
target.
- Storage: 8,000 GB times $0.001 per GB-month is $8 per month, or $96 per year, or $960 over 10
years. - Retrieval: 8,000 GB times $0.02 per GB is $160 per full-dataset restore event.
Break-even. Setting the two ten-year totals equal: 960 plus 160 times N equals 19,360, so N
equals about 115 restore events over 10 years, roughly 11.5 per year, close to monthly. Below that
frequency, approach B is cheaper; above it, approach A is cheaper. At a more realistic frequency
for a legal-hold or audit archive, say 3 full-dataset restores across the entire 10-year hold,
approach B totals 960 plus 3 times 160, or $1,440, versus approach A's $19,360, roughly 13 times
cheaper.
Qualitative factors
- SLA margin, not just average cost. Archive tiers commonly offer a faster, pricier retrieval
option and a slower, cheaper one. For a 48-hour requirement, choose whichever tier's stated
retrieval window leaves real margin under 48 hours, not one whose window is itself close to 48
hours, since that leaves no time for locating the right data, transferring it, and validating it
before the deadline. - Regulatory mandates can override cost. Some compliance regimes require genuinely
geographically redundant, continuously available copies as a requirement in itself, independent
of how often the data is actually restored. Check whether that applies before running any cost
model; if it does, approach A may be mandated regardless of the comparison above. - Tail risk of a single archive copy. If approach B's single region is lost entirely before a
restore happens, and the archive tier does not itself provide independent geographic redundancy
beyond its own region, approach B carries more risk of total data loss than approach A's
two-region design. This risk does not show up in the average-cost model and has to be weighed
separately.
Variables that determine the better approach
Dataset size, retention duration, the price differential between the fast and cold tiers, expected
restore frequency over the retention horizon (the single largest lever, per the break-even
calculation), the archive tier's actual retrieval latency relative to the SLA's margin, and any
hard regulatory requirement for geographic redundancy that removes the choice from a pure cost
comparison.
How to model cost and risk
Compute storage cost as footprint in GB times price per GB-month times months retained, summed per
copy, and retrieval cost as expected number of restores times footprint times price per GB
retrieved, comparing both approaches over the identical retention horizon and dataset size so
nothing is compared across mismatched bases. Score the qualitative risks, SLA margin and single-copy
exposure, separately from the dollar figures, and check whether a hard compliance requirement
removes the choice before optimizing the remaining trade-off on cost alone.
That is every published Backup and Disaster Recovery question for Systems Engineer so far. Browse the other topics in this category, or practice this one interactively.