Backup and Disaster Recovery Questions
Keeping data durable and recoverable when systems fail: backup design (full, incremental, differential, and snapshot strategies; point-in-time recovery), backup verification and restore testing, retention and archival policy (including compliance retention and legal holds), encryption and key management for backups, and disaster-recovery planning measured against recovery-time and recovery-point objectives (RTO/RPO). Tests whether a candidate can design a backup strategy that actually restores, choose the right retention tier for a business's downtime and data-loss tolerance, and operate backup systems safely under failure, compliance, and ransomware threats. Distinct from code-level fault-tolerance patterns (circuit breakers, retries, bulkheads) and multi-region failover architecture, which belong to high-availability-and-disaster-recovery.
Explain the difference between filesystem or volume snapshots and traditional backups. Discuss scenarios where snapshots are sufficient (e.g., rapid rollback) and cases where snapshots are not a replacement for backups (e.g., offsite, long-term retention, or provider failures).
Sample Answer
Direct answer
A snapshot is a point-in-time reference inside the same storage system it was taken from, typically
recording only what has changed since the snapshot point and depending on the original data still
being present as its base. A backup is an independent, complete copy stored in a separate location
or system, with no ongoing dependency on the original surviving. That single distinction, where the
data physically lives relative to the source, decides everything about what each one can and cannot
survive.
Structured elaboration
When snapshots are sufficient. Rapid rollback is the clearest case: undoing a bad deployment,
a failed patch, or an accidental change within minutes, where near-instant reversal matters more
than surviving a total loss of the storage system itself. Snapshots are typically much faster to
create and restore from than a full backup, since they do not move or copy the full dataset, only
the changes since the reference point, which is also what makes them cheap enough to take very
often, hourly or more, as a frequent safety net between less frequent full backups.
Offsite protection. A snapshot that lives on the same storage array or system as the original
data provides no protection if that array, its site, or its data center is lost, damaged, or
compromised, a fire, a flood, or ransomware that reaches the array's own management plane. A backup
stored somewhere genuinely independent, a different system, a different location, ideally a
different administrative boundary, protects against exactly this class of failure, which a snapshot
structurally cannot, because it never leaves the source system in the first place.
Long-term retention. Because most snapshot implementations rely on tracking changes relative to
the live volume, retaining very old snapshots for years is usually neither practical nor storage
efficient: the underlying change-tracking metadata and retained delta chains grow large and
unwieldy over long periods, and many snapshot systems have practical limits on how many, or how old,
snapshots they can efficiently keep. Backups, especially on a cheap archive tier, are the
appropriate mechanism for retention measured in months to years rather than days.
Provider or system failures. If the failure is the storage platform itself, a hardware fault, a
bug in the storage controller, or, in a cloud context, a regional outage or account-level issue
with the provider hosting the snapshots, then any snapshot stored on or by that same system is
unavailable right along with everything else. Only a genuinely independent backup copy, a different
system and ideally a different provider, account, or region boundary, survives a failure of the
platform that hosted both the original data and its snapshots.
Why real disaster-recovery strategy uses both
Snapshots handle fast, frequent, cheap local recovery, the "undo that" case. Backups handle genuine
disaster recovery, offsite protection, and long-term retention, the "the building is gone" or "we
need this from three years ago" case. Relying on snapshots alone is a common and genuinely
dangerous shortcut precisely because snapshots look like backups, both let you go back to an
earlier point, while having a structurally different failure-survival profile: a snapshot survives
everything except losing the system it lives on, and losing that system is exactly the scenario a
backup exists for.
Design a monitoring and alerting plan for backup systems: what would tell you a backup silently failed, and what would you put in front of an on-call engineer versus what a manager needs to see on a dashboard?
Sample Answer
Direct answer
A silent backup failure is one where the job scheduler reports success, or reports nothing at all,
while the actual outcome is wrong, so detecting it needs signals that are independent of the job's
own self-reported status. What goes in front of an on-call engineer and what goes on a manager's
dashboard should differ, because the two audiences need to do different things with what they see:
one needs to act right now, the other needs to judge overall posture. The right level of tooling to
build any of this with scales with team size; the underlying questions do not.
Structured elaboration
Signals that reveal a silent failure, from basic to more advanced.
A small team with limited tooling can start with a short, cheap checklist and it is a legitimate,
cost-appropriate answer, not a lesser version of a larger build:
- Missing-job detection. Alert if a job that is supposed to run on a schedule simply did not
run, or reported nothing at all, by its expected completion time, sometimes called a dead man's
switch. This catches failures upstream of the job even starting, a down scheduler, a
disconnected agent, which a "job failed" alert alone would never catch, since the job never got
the chance to fail. - Size or duration anomaly. Alert if a backup's output size or duration is drastically
different from its recent normal range. A backup that usually writes 50 GB and one night writes
500 MB most likely did not actually capture the data it thinks it did, even though the job
reports success. - Basic capacity threshold. Alert on backup storage utilization crossing a defined threshold,
the same check a small team might otherwise only do manually on a weekly review, turned into an
automated alert.
A team with more monitoring maturity adds depth on top of the same three ideas:
- Checksum and catalog completeness results feeding an alert, rather than only the job
scheduler's own status. - Automated restore or synthetic verification results treated as a first-class monitored
signal, with its own alert if a check fails or has not run recently. - Trend-based alerting, not just threshold-based, catching statistically meaningful drift, for
example backup duration creeping upward over weeks, which often precedes an outright failure,
rather than waiting for a hard threshold to be crossed.
What goes in front of an on-call engineer. Actionable, specific, real-time information: which
job or system failed or looks anomalous, what kind of failure it is (missed schedule, suspicious
size, verification failure), a direct link or runbook to the remediation steps, and enough context
to triage severity, a single low-priority file share versus a production database, without having
to dig for it. Filter aggressively: alert fatigue is a real risk, so on-call should see things that
need action now, not every warning-level signal.
What a manager needs to see on a dashboard. Aggregate, trend-level information, not per-job
noise: overall backup success rate over a time window, stated clearly as a percentage of scheduled
jobs, not of all systems, since those are different denominators and easy to conflate; restore-test
pass rate, arguably more important than raw job success rate, since a high job-success rate alone
does not prove data is actually recoverable; open or unresolved incidents and their age; and
capacity or cost trending, whether a budget increase is coming. The manager view answers "should I
be worried about our overall recoverability posture." The on-call view answers "what do I need to
fix right now."
How this scales with team size
For a small shop, the entire monitoring system might legitimately be a short checklist, someone
glancing at the basic-tier signals above on a regular cadence, plus a simple weekly rollup for
whoever is the de facto backup owner, rather than a purpose-built platform, and that is a
reasonable choice at that scale. For a larger team with dedicated tooling, the same underlying
questions, detecting silent failure, giving on-call and management different views, get answered
with dedicated monitoring, automated verification pipelines, and separate dashboards. The
principles do not change between the two; only the tooling maturity applied to them does.
Compare snapshotting and replication as data protection approaches. Explain at a high level how each works, the typical consistency guarantees they provide, common use cases, and limitations when used for long-term retention and quick recovery.
Sample Answer
Direct answer
Snapshotting and replication protect against different failure classes and are not substitutes for each other: a snapshot is a point-in-time copy that's good for rolling back a mistake on the same storage system, and replication is a continuously-updated copy that's good for surviving a whole site or region going down, and neither one alone is a substitute for an independent, retained backup.
Structured elaboration
Snapshotting: how it works. A snapshot captures the state of a volume, filesystem, or database at a specific instant, typically using copy-on-write or redirect-on-write: the storage system doesn't copy all the data up front, it starts tracking which blocks change after the snapshot was taken and preserves the pre-change version of any block that gets overwritten, so a snapshot is cheap to create and initially consumes little extra space.
Snapshotting: consistency guarantees. By default, a snapshot is crash-consistent: the captured state looks like what you'd see after an unexpected power loss, internally coherent at the storage layer but with no guarantee that an application's in-memory or in-flight work was flushed to disk first. Getting an application-consistent snapshot (one that also captures a coherent application-level state, such as a database with no half-written transaction) requires the application's cooperation: briefly pausing writes, flushing buffers, or using a storage-integrated hook the application supports.
Snapshotting: use cases. Fast, frequent recovery points for operational mistakes (a bad deployment, an accidental delete, a botched migration) where you want to roll back to "ten minutes ago" cheaply and quickly, all on the same storage system.
Snapshotting: limitations for retention and recovery. A snapshot usually depends on the same underlying storage volume it was taken from; if that storage system or site is destroyed, the snapshot is destroyed with it, so a snapshot alone offers no protection against a storage-system or site-level failure. Snapshot chains can also grow unwieldy over long retention periods: many storage systems either cap how many snapshots they'll retain gracefully or see rising overhead as the chain lengthens, so snapshots are usually not the right primary mechanism for months- or years-long retention.
Replication: how it works. Every write (or a stream of change records) is continuously copied from a primary to one or more replicas, either synchronously (the write isn't acknowledged to the client until the replica confirms it too) or asynchronously (the write is acknowledged immediately and the replica catches up shortly after).
Replication: consistency guarantees. Synchronous replication guarantees zero data loss for any write the primary acknowledged, at the cost of added write latency (every write waits on a round trip to the replica). Asynchronous replication has lower write latency but carries a small, real window of potential data loss equal to whatever the replication lag is at the moment of a primary failure.
Replication: use cases. High availability failover and near-real-time disaster recovery to a different site or region, where the goal is a very short recovery time and a very small recovery point gap if the primary goes down.
Replication: limitations for retention and recovery. Replication faithfully propagates everything, including mistakes: an accidental delete or logical corruption on the primary replicates to the replica within seconds, so replication provides no protection at all against a logical or application-level error, only against physical infrastructure loss. It also isn't a retention mechanism on its own; a replica reflects the current state, not a history of past states, so "restore to how things looked 90 days ago" isn't something replication by itself can do.
Worked example
An e-commerce platform's order database uses synchronous replication to a standby in a second availability zone (protecting against a rack or zone failure with effectively zero data loss for acknowledged writes) plus hourly application-consistent snapshots retained for 7 days (protecting against an operator accidentally truncating a table). One afternoon, a deploy script runs an unintended bulk update that corrupts several thousand order rows. Replication faithfully copies the corrupted rows to the standby within moments, so failing over to the replica does not help: the standby has the same corrupted data. The snapshot from an hour before the deploy, however, reflects the state before the mistake, and the team restores the affected tables from that snapshot. The two mechanisms solved two different problems: replication would have saved them from a zone outage; only the snapshot saved them from the logical error.
Trade-offs and pitfalls
- Treating replication as "our disaster recovery plan" without a separate retained backup or snapshot leaves an organization with no defense against logical corruption or accidental deletion, which is a more common cause of real data-loss incidents than a full site failure.
- Crash-consistent snapshots are cheaper and simpler but can leave a database in a state that requires its own crash-recovery process to become usable (replaying its transaction log from the crash-consistent point); application-consistent snapshots avoid that but cost more (briefly pausing or quiescing the application) to take.
- Synchronous replication's zero-data-loss guarantee comes with a real latency cost on every write, which is why it's typically used only for nearby sites (where the round-trip cost is small); at longer distances, most systems fall back to asynchronous replication and accept a small, monitored replication-lag window instead.
Compare an 'incremental-forever' backup model with a traditional incremental model that includes periodic full backups. Discuss the impacts on restore time, storage efficiency, catalog complexity, and operational overhead, and recommend scenarios where one approach is preferable.
Sample Answer
Direct answer
The real difference is what a "restore chain" looks like under each model: how many backup sets
have to be applied, in sequence, to reconstruct data as of a given point in time. Traditional
incremental (periodic full plus daily incrementals) resets that chain to a length of one every
time a new full runs, at the cost of periodically re-writing the entire dataset. Incremental-forever
takes one full copy ever, then only ever stores changes, and relies on the backup software to
assemble a usable restore point from the accumulated incrementals (often by periodically merging
them into a synthetic "virtual full" behind the scenes) rather than the operator ever seeing a
literal second full backup.
Structured elaboration
Restore time. Under traditional incremental with a weekly full, restoring to a point six days
after the last full means applying the full plus all six intervening incrementals, a chain of
seven backup sets. Restore time and risk both grow the further you are from the last full: more
sets to apply, more opportunities for one link in the chain to be missing or corrupt. Under
incremental-forever, the software typically keeps a synthetic full current in the background, so
restoring to nearly any recent point applies that single synthetic full rather than a growing
chain, keeping restore time comparatively flat regardless of how long it has been since the
original full.
Storage efficiency. Traditional incremental re-writes the entire dataset at every full backup,
even the large majority of it that has not changed since the last one. Incremental-forever only
ever stores actual changed data after the initial full, which is more storage-efficient over time,
but the software still needs to periodically read through and consolidate the accumulated
incrementals to build each new synthetic full, which is a real, recurring compute and I/O cost
even though it does not require re-reading the original source system.
Catalog complexity. Traditional incremental's catalog only has to track a relatively short
chain (the current full plus the incrementals since it), so a corrupted Tuesday incremental only
threatens restores that specifically need Tuesday's exact state. Monday's incremental and the
underlying full are still independently valid, and next week's full resets the risk entirely.
Incremental-forever's catalog has to track potentially years of block-level lineage with no
periodic reset, so losing or corrupting the catalog, or a single unmerged incremental before it
gets folded into a synthetic full, is a materially larger and longer-lived risk. This catalog
fragility is the main operational cost of the incremental-forever model.
Operational overhead. Traditional incremental concentrates load into a periodic spike: a full
backup of a 10 TB system means moving roughly 10 TB of I/O and network traffic in one maintenance
window, however often that full is scheduled. Incremental-forever spreads load evenly across
days, avoiding that spike, but requires ongoing background merge and consolidation jobs plus
periodic integrity checks to confirm the never-refreshed logical full is still valid, since there
is no natural, periodic full re-read to catch quiet data corruption the way the traditional
model's next full backup would.
Worked example
Take a 10 TB dataset with a daily change rate of roughly 2% of the dataset (a labeled example
assumption, not a universal constant). Under a traditional weekly-full model, the full backup
writes close to 10 TB, and each of the six following daily incrementals writes roughly 200 GB
(2% of 10 TB), for about 1.2 TB of incremental data across that week. A restore taken on day six
must apply the 10 TB full plus all 1.2 TB of incrementals, seven backup sets in sequence, meaning
a single corrupted incremental anywhere in that chain can block the whole restore. Under
incremental-forever, the initial full is a one-time 10 TB read, after which only roughly 200 GB
per day is ever captured, and a restore on any given day pulls from the current synthetic full
rather than re-walking a growing weekly chain, at the cost of needing to trust that the
background merge process itself has run correctly and recently.
Recommendation
Favor traditional incremental with periodic fulls for smaller datasets, shorter retention windows,
or environments where a fresh, independently verifiable full backup on a predictable cadence
matters more than storage cost, since each new full effectively resets both storage growth and
catalog risk to zero. Favor incremental-forever for very large datasets where re-reading a full
copy on a recurring basis is itself operationally expensive (large virtual machine estates, large
file shares), where storage and bandwidth cost dominate the budget, and where the backup product's
synthetic-full generation and catalog integrity checking are mature enough to trust without a
periodic literal full backup as a safety net.
Problem-solving: Given a constrained backup budget, describe how you would schedule backups and retention for different data classes (transaction logs, databases, large file shares, VM images) to balance cost and business risk. Provide concrete suggestions (frequency, storage tier, and retention) and explain trade-offs.
Sample Answer
Direct answer
Do not optimize each data class in isolation. Model total cost across all of them, then push the
biggest, lowest-risk class as aggressively as possible toward a cheap tier and short retention,
because that is where a constrained budget actually gets freed up, and spend the savings on
keeping the small but critical classes on a fast tier with a tight schedule.
Structured elaboration and concrete plan
- Transaction logs. Frequency: continuous or near-continuous shipping, or every 5 to 15
minutes for systems that cannot ship continuously. Storage tier: standard or hot, since logs
exist specifically to support fast, fine-grained point-in-time recovery alongside a full or
differential backup. Retention: short, roughly 7 to 14 days, since logs are only useful in
combination with the full and differential chain they support; keeping logs long after the
backups they pair with have expired protects nothing. - Databases (the full and differential backups themselves). Frequency: nightly full or a
weekly full with nightly differentials. Storage tier: standard while recent, moving older
restore points to a cheaper tier as they age. Retention: roughly 30 to 90 days for operational
recovery, longer only where compliance specifically requires it. - Large file shares. Frequency: daily incremental with a weekly full, or an incremental-forever
model given the typical size. Storage tier: cool or infrequent-access, since restores of file
shares are comparatively rare. Retention: 30 to 90 days of operational retention, with an
optional monthly or yearly snapshot moved to a cold archive tier for anything that needs to be
kept longer at a much lower ongoing cost. - VM images. Frequency: daily image-level backup. Storage tier: standard for only the most
recent restore point (fast recovery after a bad deploy or patch), cold or cool for older restore
points. Retention: a grandfather-father-son style scheme, for example daily for one to two weeks,
weekly for a couple of months, monthly for a year.
Worked example: where the budget actually goes
Take illustrative footprints (the resulting storage under the retention choices above, not a
separate derivation): databases 5,000 GB, transaction logs 200 GB, file shares 50,000 GB, VM images
20,000 GB, for 75,200 GB total. Using illustrative prices of $0.02 per GB-month on a standard tier
and $0.01 per GB-month on a cool tier (order-of-magnitude realistic ratios, not live vendor
pricing), keeping everything on the standard tier costs 75,200 GB times $0.02 per GB-month, or
$1,504 per month.
A tiered plan instead keeps databases and logs (5,200 GB) on standard, since they need fast
restores: 5,200 times $0.02 is $104 per month. File shares (50,000 GB) move entirely to the cool
tier: 50,000 times $0.01 is $500 per month. VM images split, with roughly the most recent
restore point (about 1,400 GB) staying on standard for fast recovery and the remaining 18,600 GB
on cool: 1,400 times $0.02 is $28, plus 18,600 times $0.01 is $186, for $214 per month.
Total under the tiered plan: 104 plus 500 plus 214, or $818 per month, versus $1,504 per month if
everything stayed on standard, a saving of $686 per month, roughly 46%. Of that $686 in savings,
$500, about 73%, comes purely from moving the file shares to the cool tier. That is the point: file
shares are the largest volume and the lowest per-byte business risk of the four classes, so tiering
them is the single highest-leverage lever, and it is what funds keeping the small, high-value
database and log tiers fast and frequent.
Trade-offs
Cheaper tiers add retrieval latency and sometimes retrieval fees. The transaction logs and the most
recent database and VM restore points must not be pushed to a cold tier even though it is cheaper,
because the entire point of frequent logging and recent-point backups is fast recovery; a
budget-constrained plan protects recovery time and recovery point for the highest-risk classes
first, then applies cost savings to the classes that can genuinely absorb slower restores.
Unlock Full Question Bank
Get access to all Backup and Disaster Recovery interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.