Large-Scale Infrastructure Operations Questions

SLI vs SLO vs SLA (concise):

• SLI (Service Level Indicator): a measured metric representing user experience (e.g., p99 latency, successful request rate).
• SLO (Service Level Objective): an internal target for an SLI over a window (e.g., 99.9% of requests under 200ms).
• SLA (Service Level Agreement): a contractual commitment to customers, often with financial penalties if missed; typically built from SLOs but legally binding.

Concrete SLIs for a globally distributed web API:

1. Latency SLI: p95 end-to-end request latency measured at edge (client-to-edge-to-backend) per region.
   • SLO: p95 < 150 ms per region, measured over 30 days, with 99.9% compliance.
2. Availability SLI: successful request ratio (HTTP 2xx/3xx) measured at the API gateway.
   • SLO: 99.99% successful requests over 30 days (≈4.38 minutes downtime/month).

Operational actions when error budget is exhausted (organization-wide):

• Immediately throttle non-essential work: halt launches, feature rollouts, experiments, and large migrations until budget is restored.
• Shift to reliability-focused work: prioritize incidence remediation, root-cause fixes, capacity increases, rollback risky changes.
• Increase monitoring & alerting sensitivity and run targeted game-days to reproduce/fix.
• Require change freezes and stricter change approvals (e.g., canary percentage limits, staged rollouts).
• Communicate to stakeholders and adjust customer expectations; if SLA risk exists, notify legal/PM and prepare mitigation/compensation plans.
• After recovery, run a blameless postmortem and update SLOs, runbooks, and automation to prevent recurrence.

Requirements & constraints

• Data: 5 PB global object store
• RPO: 1 hour (max 1 hour of data loss)
• RTO: 6 hours (service restored within 6 hours)
• Multi-region deployment, must plan bandwidth, verification, and non-disruptive restore testing

High-level approach

• Continuous incremental replication of object-level changes + hourly metadata/snapshot checkpoints. Keep full “archive” snapshots (manifest + objects pointer) daily/weekly for long-term retention.
• Tiered restore: metadata/manifest first, then hot objects (priority lists), then warm/cold background restore in parallel to meet RTO.

Snapshotting cadence & metadata

• Hourly checkpoints: capture object manifest, versions, ETags/checksums, and change-log offsets. These are small and allow RPO=1h.
• Daily incremental “synthetic fulls”: materialize the incremental chain into a consolidated manifest to speed restores.
• Weekly archive: immutable manifests + retention markers for compliance.

Incremental replication model

• Event-driven replication: every object create/overwrite/delete writes to an append-only op-log (CDC). Replication consumers stream ops -> target region(s).
• Use object versioning plus ETag checksums; replicate object bodies only when checksum changed.
• Multipart uploads: replicate completed parts or use resumable replication pointers to avoid retransfer.
• De-duplication & compression on transport (chunk-level dedupe, Brotli/flate) to reduce bandwidth.

Bandwidth planning (how to size links)

• Formula: required_bandwidth_gbps = (expected_changed_data_per_hour_bytes * 8) / 3600 / 1e9
• Example scenarios:
  • If 0.1% churn/hour of 5 PB = 5 TB/hour → bandwidth ≈ (51024 GB8)/3600 ≈ 11 Gbps
  • If worst-case 1% churn/hour = 50 TB/hour → ≈ 111 Gbps
• Plan for headroom (1.5–2×) and peak windows. Use:
  • Dedicated WAN links (MPLS/Direct Connect/Cloud Interconnect)
  • Traffic shaping: prioritize op-log & small metadata, background bulk only in off-peak
  • Parallel streams, pipelining, and backpressure
  • Cross-region peering + CDN edge caches for hot restores
  • Optionally use physical transfer for full re-seed (snowball) for rare large-scale restores

Restore plan to meet RTO

• Prioritization:
  1. Metadata + manifests (minutes)
  2. Application-critical buckets/objects (hot list) — restore in parallel
  3. Remaining objects in background
• Parallelization: many worker nodes pulling objects concurrently; restore workers honor bandwidth limits.
• Use local caching and tiered storage: restore hot objects into SSD-backed cache for immediate service.
• Targeted RTO math: calculate per-object restore throughput required = total_size_to_restore / available_restore_time; add concurrency accordingly.

Restore verification & integrity

• Verify checksums (ETag/hash) after transfer.
• Inventory reconciliation: compare manifests vs. restored objects; reconcile deletes and versions.
• End-to-end test: sample file application-level verification (e.g., open/parse).
• Maintain immutable audit logs of restore operations and checksums.

Testing restores without impacting production

• Staged recovery environments: isolated test clusters with identical software and scaled-down data sets.
• Canary restores: restore critical small-bucket subsets to test infra and automation end-to-end.
• Use synthetic datasets or masked copies of production metadata/objects for full-scale rehearsals.
• Shadow restores: perform restores to a parallel region that doesn’t affect live traffic; run health checks and smoke tests.
• Automated recovery drills: scheduled weekly/monthly runbooks executed in low-impact windows; measure metrics (time to metadata, time to first hot-object, data integrity).
• Chaos testing: simulate partial network outages and verify replication catch-up and restore resilience.

Operational & monitoring

• SLOs: replication lag < X seconds, op-log commit RTT, restore throughput targets.
• Alerts on backlog in op-log, replication failure, checksum mismatch rates, bandwidth saturation.
• Automation: orchestration for failover/restore, throttling controls, and runbook-driven escalations.

Trade-offs & considerations

• Aggressive hourly snapshots/continuous replication increases cost and bandwidth but meets RPO.
• Use of dedupe/compression reduces bandwidth but increases CPU.
• Physical re-seed reduces WAN cost for initial large transfers but slower for incremental catches.

This design balances continuous small-change replication for RPO=1 hour with prioritized, parallel restores and verification to meet RTO=6 hours while enabling safe, regular restore testing via staged and synthetic restores.

Infrastructure as Code (IaC) is the practice of defining and managing infrastructure (networks, compute, storage, policies) through declarative or programmatic configuration files so provisioning, versioning, and automation are reproducible, reviewable, and testable—key for SREs to ensure reliable, auditable, and repeatable operations.

Comparison:

• Multi-cloud / Hybrid

  • Terraform: Strong multi-cloud first-class support via many providers; good for hybrid setups (on‑prem + cloud) and consistent workflow across platforms.
  • CloudFormation: Native to AWS; excellent for deep AWS features but poor for multi-cloud (use for AWS-only stacks).
  • Pulumi: Multi-cloud capable and uses real programming languages, making complex orchestration across clouds easier but requires language ecosystem management.

• State management

  • Terraform: Central concept of state file; supports remote backends (S3+Dynamo/Consul, Terraform Cloud) with locking—robust but requires careful locking/secret handling.
  • CloudFormation: No separate state file; AWS manages stack state in service which avoids local state hassles but limits cross-account orchestration patterns.
  • Pulumi: Maintains state (Pulumi Service or self-hosted backends) with stack concepts; similar operational needs to Terraform but with richer metadata.

• Modularization

  • Terraform: Modules are first-class; encourages reusable module registries and versioning—good for org-wide standards.
  • CloudFormation: Nested stacks and macros work, but modules are less ergonomic; AWS CDK (higher level) improves reuse within AWS.
  • Pulumi: Uses language-native packages and modules — very flexible for complex logic and reusable components, but requires engineering discipline for API stability.

• Fit for large-scale infra & automated SRE workflows

  • Terraform: Best balance for large, multi-team environments—stable, predictable plan/apply lifecycle, policy-as-code integrations (Sentinel, OPA), CI/CD pipeline friendly. SREs benefit from clear drift detection and remote state locking.
  • CloudFormation: Best when AWS-only and you need deep service feature support and tight AWS integration (StackSets, change sets). Simpler state handling reduces ops overhead but multi-account orchestration can be clunkier.
  • Pulumi: Great for complex logic, custom orchestration, and embedding IaC into existing engineering workflows; fits SREs who prefer imperative logic and testing with real languages. Requires language/tooling governance at scale.

Recommendation (SRE lens): Use Terraform for multi-cloud consistency and team-scale governance; CloudFormation (or AWS CDK) when you’re AWS-centric and need native features; choose Pulumi when you need advanced programmatic constructs and your team can manage language-based pipelines and library/versioning discipline. In all cases, enforce remote state, locking, CI-driven plans with automated policy checks, testing, and least-privilege credentials to integrate into SRE automated workflows.

Approach: compare the three options across five dimensions you asked for (cache-control, invalidation, origin shielding, cost vs latency, ops complexity) and conclude with recommended trade-offs for an SRE.

Third‑party CDN (Akamai/Cloudflare)

• Cache-Control: rely on long max-age + immutable for versioned content; use stale-while-revalidate for UX.
• Invalidation: fast global purge APIs, but can be rate‑limited / costed for mass invalidations. Prefer cache-busting (fingerprinted URLs) for large-scale static.
• Origin shielding: built‑in shielding/PoP hierarchy; you get extra protection for origin at no ops cost.
• Cost vs latency: typically lowest latency (many PoPs) but egress costs can still be significant depending on plan—often negotiable.
• Ops complexity: minimal; SLA, DDoS, analytics included. Good for rapid rollout.

Cloud‑native CDN (Cloud Provider CDN like Cloud CDN / Azure Front Door)

• Cache-Control: same headers apply; integrates tightly with object storage lifecycle and signed URLs.
• Invalidation: usually simple via SDK/CLI; might be regional propagation delays.
• Origin shielding: often automatic when colocated with provider storage (e.g., GCS/S3 + CDN) — reduces egress between origin and edge.
• Cost vs latency: cheaper egress within same cloud; latency good but fewer global PoPs than top-tier CDNs.
• Ops complexity: moderate; benefits from unified IAM, logging, and billing. Easier to automate infra-as-code.

Build your own edge POP cache (self-hosted caches in colo or cloud edge VMs)

• Cache-Control: you must strictly enforce headers at edge; implement TTL, revalidation, and conditional GETs (If-Modified-Since/ETag).
• Invalidation: complex — need distributed purge protocol or rely on short TTLs / versioned keys; mass invalidation is operationally heavy.
• Origin shielding: you must design cache hierarchy (regional cache → origin), adding complexity but possible.
• Cost vs latency: potential lowest egress spend if you negotiate bandwidth or colocate caches; latency depends on PoP density — expensive to match global coverage.
• Ops complexity: high — provisioning, monitoring, security, DDoS mitigation, consistency and cold-cache warming are your responsibility.

SRE recommendations

• If time-to-market, reliability, and low ops overhead matter: pick third‑party CDN; negotiate egress in contract.
• If most traffic is from cloud-hosted buckets and you want lower egress bills + simpler infra: use cloud‑native CDN.
• Build-your-own only if you have predictable heavy traffic, existing colo relationships, and a team to run global cache fleet; otherwise risk high ops cost.
• Always: version static assets for immutable caching, use cache-control/ETag for dynamic revalidation, enable origin shielding, implement cache-warming for releases, and measure egress vs latency with real traffic tests before large rollouts.

Requirements & constraints:

• Full automated failover drill for critical services with minimal customer impact
• Synthetic verification + data integrity checks
• Scheduled + ad-hoc runs, isolation of test traffic, automated validation, rollback on failure
• Auditability for compliance, RBAC

High-level architecture:

• Orchestrator (Kubernetes Job/Argo Workflows or Jenkins/Concourse pipeline) drives drill steps
• DR Playbook service (YAML/JSON per-service drill definitions)
• Environment controller (provisions isolated DR namespace/tenants, test network overlays)
• Traffic switcher (feature-flagged DNS / traffic steering via Envoy or global load balancer)
• Synthetic test runner (k6/Robot/pytest suites) and data-integrity checker (schema checks, checksums, row counts, test DB clones)
• Observability & validation engine (Prometheus + custom validators) reads SLOs, metrics, logs
• Audit & reporting (immutable logs to S3 + signed attestations)

Drill flow:

1. Pre-checks: capacity, backups, schema compatibility, consent windows (business calendar)
2. Snapshot production data to isolated test copies (snapshots, logical backups) with PII masking
3. Provision isolated DR environment: separate VPC/subnets, namespaces, service accounts
4. Controlled failover: orchestrator triggers traffic switcher to route a small percentage (canary) of synthetic traffic only to DR env; production traffic untouched
5. Synthetic verification: health endpoints, end-to-end flows, latency/throughput tests run as suites; data-integrity: replay subset of write workload against test DB and verify checksums/consistency
6. Progressive ramp: if green, increase percentage and run full failover simulation; if any validator fails, trigger rollback

Isolation of test traffic:

• Use separate DNS names / header-based routing or mTLS client certs to ensure synthetic requests are distinguishable
• Network-level isolation via test VPC peering and egress controls
• No production write paths enabled from test env unless using read-only snapshots

Automation & validation:

• Validators codified as pass/fail with thresholds; observability exposes SLOs and slis
• Orchestrator polls validators and enforces timeouts
• Failure modes captured: critical, warning with remediation playbooks
• All actions logged with timestamps, operator IDs, CI run IDs for compliance

Rollback & minimizing customer impact:

• Default to "non-disruptive" canary-first approach; only escalate to full traffic switch after multi-stage green checks
• On failure: immediate traffic re-route back to production via traffic switcher (atomic DNS/ratelimit rollback), terminate DR env to prevent drift, and run automated remediation (restore snapshots, re-sync)
• Post-failure: run automated incident creation with attachments, and require manual approval for repeat attempts

Compliance & audit:

• Immutable logs, signed attestations, encryption-at-rest/in-transit, PII masking, role-based approvals, retention policies
• Pre-drill approvals and blackout windows recorded; drill reports with evidence (screenshots, test results, hashes) stored for audits

Trade-offs:

• Full production write simulation expensive and risky — prefer sampled replay onto masked clones
• Faster drills vs deeper tests: balance via tiering of services (critical vs non-critical)

Implementation notes:

• Use Infrastructure-as-Code (Terraform) and GitOps for playbooks
• Use feature flags and canary tooling (Flagd/LaunchDarkly + Istio/Envoy)
• Continuous improvement: run quarterly full drills, weekly mini-canaries, and automated postmortems.