Backup Strategy and Operations Questions

Designing, implementing, and operating reliable backup systems that meet business recovery objectives. Topics include deciding what to back up and why, data classification and tiering, mapping Recovery Time Objective and Recovery Point Objective to backup types and cadence, backup frequency and windows to minimize production impact, retention policy design and lifecycle management, storage selection from local fast restores to off site or cloud tiered storage and air gapped copies, deduplication and compression trade offs, application aware backups and quiescing, automation and orchestration of backup workflows, monitoring and alerting for backup health, integrity verification and routine restore drills, encryption and access controls for backup data, cost and capacity planning, and planning restores at scale. Candidates should also be prepared to discuss trade offs, operational runbooks, and how backup operations support compliance requirements.

HardTechnical

88 practiced

As a senior systems administrator you must recommend whether to purchase a commercial backup platform or build an internal backup solution for a global enterprise. Create a decision framework covering total cost of ownership, SLAs, feature gaps, vendor lock-in, security and compliance, operational overhead, and a final recommendation process including pilots and evaluation criteria.

Sample Answer

**Situation & Objective** As the senior systems administrator I must decide build vs buy for global enterprise backups. I propose a decision framework covering TCO, SLAs, features, lock-in, security/compliance, ops overhead, and a pilot/evaluation plan.

**1) Total Cost of Ownership (TCO)** - Capital: hardware, datacenter, network upgrades, site-to-site replication. - Ongoing: software dev/maintenance, licenses, support FTEs, storage egress, cloud snapshot costs, power/cooling. - Opportunity: time to market, delayed DR readiness. - Quantify 3–5 year NPV for both options.

**2) SLA & RTO/RPO Matching** - Map business-critical apps to required RTO/RPO. - Assess vendor’s contractual RTO/RPO, SLA credits, and historical uptime vs in-house achievable metrics.

**3) Feature Gap Analysis** - Required: incremental/differential backups, global deduplication, compression, WAN acceleration, catalog/search, granular restores, cross-region replication, tested DR orchestration, immutability. - Rate vendor vs engineered solution and list gaps.

**4) Vendor Lock-in & Portability** - Evaluate export formats, APIs, standard protocols (S3, NFS), portability risk, exit cost. - For buy: require data export SLA and tools to migrate off platform.

**5) Security & Compliance** - Encryption at-rest/in-transit, KMS integration, key management, role-based access, MFA, WORM/immutability for regulatory holds (GDPR, HIPAA, SOX), audit logs, SOC2/HIPAA certifications. - Map controls to compliance gaps.

**6) Operational Overhead** - Staffing: 24x7 on-call, patching, DR runbooks, backup verification (automated restores), monitoring/alerting, vendor management. - Estimate FTEs and skills training.

**7) Recommendation Process, Pilot & Evaluation** - Shortlist 2–3 vendors + internal MVP. - Pilot scope: 10 critical workloads across 3 regions for 8–12 weeks. - Evaluation criteria (weighted): SLA reliability 20%, RTO/RPO compliance 20%, feature coverage 15%, TCO delta 15%, security/compliance 15%, operational effort 10%, exit portability pass/fail. - Success thresholds: meet RTO/RPO in 95% of tests, successful full-site failover dry-run, restore verification success rate > 99%, demonstrated data export within SLA.

**Decision Rule** Choose commercial if pilot shows >10–20% lower TCO (3yr) and meets SLA/security/portability thresholds and reduces ops headcount or risk. Otherwise build internal if custom requirements (extreme portability, proprietary stacks) justify higher control and long-term lower cost.

**Final note** Include contract terms for SLAs, data ownership, exportability, and a 6–12 month review clause after deployment.

HardTechnical

89 practiced

Describe and sketch a CI/CD pipeline that automatically validates backup and restore functionality: triggers, synthetic test data creation, storing test backups, performing restores in isolated environments, integrity and functional checks, and failure reporting. Indicate technologies you would use and how to ensure the pipeline is safe for production.

Sample Answer

**Approach (one sentence)** I’d build an automated CI/CD pipeline that regularly creates synthetic test data, runs backups to the same backup target used in prod, restores into isolated environments, runs integrity + functional checks, and reports failures — all gated by strict safety controls.

**High-level sketch (ASCII)**repo (backup-tests) -> CI (GitHub Actions/Jenkins) -> Provision isolated env (Terraform/Ansible) -> Load synthetic data -> Run production backup job -> Store test backup (S3/EBS/RDS snapshot) -> Restore to isolated env -> Integrity + functional tests -> Teardown + report

**Triggers**- Scheduled (cron daily/weekly) in CI- On every backup job completion (webhook from backup system)- Manual run via pipeline UI

**Steps & tools**1. Provision isolated environment - Terraform for infra (VPC, subnets, security groups), Kubernetes namespace or small EC2/RDS instance - Use separate AWS account or isolated VPC + network ACLs2. Create synthetic test data - Scripts using Faker / custom data generator; for DB use pgbench/SQL load scripts; for files use randomized file sets - Ensure data resembles production patterns (size, schema) but is anonymized3. Run backup job against test target - Trigger same backup mechanism as prod (restic, AWS Backup, Velero for k8s, enterprise backup tool) - Store artifacts to actual backup target (S3 with prefix test/, separate bucket, or dedicated backup vault)4. Register and store backup metadata - Persist checksums, manifest, and snapshot IDs in CI artifact store (or DynamoDB)5. Restore into isolated environment - Create ephemeral resources, restore from saved snapshot or object store - Ensure network isolation, no routing to prod6. Integrity checks - Verify file checksums and object counts - DB-level checksums/row counts, schema compare - Application smoke tests (API endpoints, auth flows) using Postman/Newman or pytest7. Functional tests - Run representative transactions to verify app behavior8. Report & teardown - On failure: collect logs, artifacts, diagnostics; create issue (Jira/GitHub), notify Slack/PagerDuty - On success/after tests: destroy environment to avoid drift/costs

**Safety controls for production**- Use separate AWS account or authenticated role with least privilege; CI uses short-lived credentials via Vault- Run restores only in isolated VPCs/namespaces with no routing to prod networks; deny cross-account write permissions- Anonymize/sanitize all synthetic and (if ever using masked prod samples) data; never use raw prod data- RBAC and approval gates for manual runs; resource quotas and budget alerts- Run test backups with a test prefix or separate backup vault to avoid collisions- Use immutable snapshots and versioned storage to prevent accidental overwrite- Dry-run mode and “fail-close” where pipeline stops if IAM/role misconfig detected

**Checks & metrics**- Success rate, time-to-restore, data-consistency (checksums), test coverage of critical flows- Alert on regression (e.g., restore time > threshold, checksum mismatch)

**Technologies (example stack)**- CI: GitHub Actions or Jenkins- Infra: Terraform, Ansible- Cloud: AWS (S3, EBS, RDS snapshots, Backup), or GCP/Azure equivalents- Backup tool: restic / Velero / vendor backup system- Secrets: HashiCorp Vault / AWS KMS & IAM- Test tooling: pytest, pgbench, Newman, Fortune/Faker- Observability: Prometheus+Grafana, CloudWatch, Slack & PagerDuty for alerts

This pipeline provides automated, repeatable validation of backups and restores while preventing any interaction with production systems.

EasyTechnical

121 practiced

Explain the difference between filesystem or volume snapshots and traditional backups. Discuss scenarios where snapshots are sufficient (e.g., rapid rollback) and cases where snapshots are not a replacement for backups (e.g., offsite, long-term retention, or provider failures).

MediumSystem Design

98 practiced

How would you design backup automation using Infrastructure as Code (Terraform/Ansible) so that new environments automatically receive the correct backup policies, tagging and lifecycle rules? Include how to enforce policies, test backups in CI/CD, and prevent drift.

Sample Answer

**Clarify goals**- New environments must get consistent backup policies, tags, lifecycle rules automatically.- Policies must be enforced, backups tested in CI/CD, and config drift prevented.

**High-level approach**- Implement reusable Terraform modules + Ansible roles, manage everything in Git (GitOps).- Enforce policy-as-code (OPA/Sentinel/AWS Config) and run automated backup tests in CI.

**Core components**- Terraform module "backup-base" that provisions: - Backup plans (e.g., AWS Backup vault + plan, RDS snapshots, EBS lifecycle rules) - Standard tags via variables (environment, owner, retention) - IAM roles/policies to restrict who can change retention- Ansible role "backup-verify" that: - Installs and configures agent-backed backups or cron jobs on VMs - Adds host-level tags/metadata - Provides restore scripts used in CI tests

**Enforcement**- Policy-as-code: - OPA/Rego or HashiCorp Sentinel policies to require tags, minimum retention, and disallow public backups. - AWS Config rules to detect non-compliant resources and trigger Lambda remediation or ticketing.- Enforce at PR / plan time: - Terraform Cloud / Enterprise runs with Sentinel, or pre-merge checks using tflint/terraform-compliance.

**CI/CD backup testing**- Pipeline steps: 1. Terraform plan/apply in a temporary test workspace to create minimal resource (e.g., test DB). 2. Trigger backup (or wait for scheduled), then force snapshot. 3. Use Ansible role to perform an automated restore to a temp instance. 4. Run smoke tests (connectivity, basic queries, file integrity). 5. Destroy test resources.- Use automated integration tests (Terratest for Go, or kitchen-terraform + Ansible/molecule) to validate both provisioning and restore.

**Preventing drift**- GitOps: only changes via PRs. No manual console edits.- Periodic drift detection: - Scheduled Terraform plan runs; alert on diffs. - AWS Config / Azure Policy to report and auto-remediate common drift (reapply tags, reset lifecycle).- State safety: remote state storage with locking (S3 + DynamoDB / Terraform Cloud).- Immutable patterns: treat backups and retention as managed only by IaC; restrict IAM to prevent manual overrides.

**Example snippets**- Terraform module inputs: retention_days, tags, backup_window.- CI test: use Terratest to create DB, create snapshot, restore, run SQL assertion, destroy.

**Why this works**- Reusable modules + Ansible ensure consistency on provision and host config.- Policy-as-code enforces guardrails pre-merge and at runtime.- CI restore tests validate actual recoverability, not just configuration.- GitOps + scheduled checks minimize drift and provide auditable changes.

HardTechnical

76 practiced

For a sharded transactional database with thousands of shards across many nodes, outline a plan to restore database state after catastrophic loss while minimizing data loss and downtime. Include strategies for parallelism, correct restore ordering, transactional consistency, index rebuilds, and handling of external references.

Sample Answer

**Situation & goals**As a Systems Administrator I’d restore thousands of shards after catastrophic loss with objectives: minimize RPO/RTO, preserve transactional consistency, and restore referential integrity with minimal downtime.

**High-level plan**- Triage: identify healthy nodes, validated backups, WAL/redo logs, and metadata (shard map, topology, config).- Prioritize critical shards (hot shards, metadata, auth, payments) for early restore to reduce business impact.

**Parallelism & orchestration**- Use an orchestration engine (Ansible/Terraform + custom worker pool) to run parallel restores per shard group with concurrency limits per host and network IO throttling.- Group shards by backup storage location and node affinity to avoid hotspotting.- Stage 1: restore base snapshot concurrently.- Stage 2: apply WAL/redo logs concurrently up to a consistent timestamp.

**Correct ordering & transactional consistency**- Restore global metadata and cluster state first so routing and shard mapping work.- For each shard: apply snapshot → replay logs up to a global target LSN/SCN derived from cluster-wide coordination service (etcd/ZK) to achieve a consistent cut.- Use two-phase approach: restore in read-only / recovery mode, validate constraints, then flip to writable in coordinated batches to avoid split-brain.

**Index rebuilds**- Delay non-critical secondary index builds until after data and constraints are restored.- Rebuild in parallel per-shard with I/O and CPU limits; prefer online/lock-minimizing rebuilds where supported.- For large indexes, use partitioned or incremental rebuilds and prioritize indexes used by critical queries.

**External references & referential integrity**- Restore and validate external reference tables (users, tenants) early.- For cross-shard foreign keys or external systems (message queues, S3 pointers), reconcile via idempotent reconciliation jobs; mark dangling refs for application-level handling if immediate repair isn’t feasible.

**Verification & cutover**- Run integrity checks (checksum, counts, FK checks where possible), smoke tests, and sample transactions.- Cutover in waves: promote groups to writable after validation; keep telemetry and quick rollback plans.

**Post-restore**- Re-sync replication, force full backups, and run long-term consistency jobs. Update runbooks and postmortem.

This plan balances speed through parallelism and safety through ordered, coordinated replay and validation.

Unlock Full Question Bank

Get access to hundreds of Backup Strategy and Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.