Junior Cloud Engineer Interview Preparation Guide (FAANG-Standard)

Direct answer

Before your work depends on it, put a written interface contract in place (the shape of the data or API, error cases, and versioning), a single named owner on each side, and an SLA (service level agreement: the vendor's contractual uptime/response commitment) for questions and changes with a defined escalation path. Then build against a mock or stub (a fake stand-in for the real API that returns data matching the agreed contract, so your team can build and test without waiting on the real thing) that matches that contract, so a late dependency delays true integration, but doesn't block your team's progress.

Framework

Before you start building. Agree the contract explicitly (schema, error handling, versioning), name one owner per side rather than 'the team', and set an SLA for response time and change turnaround, with an escalation path if it slips.

While you wait. Build and test against a mock or stub that matches the agreed contract, so your team keeps moving. Pair it with automated contract tests, so if the mock and the real dependency drift apart, you find out at build time instead of at release.

Internal-team dependency vs external vendor dependency. The mechanics differ once the other side is a vendor rather than a team you can walk over to.

Worked example

Situation: a product depends on a vendor-managed API (for example a payments or identity provider). The vendor's contract commits to 99.5% availability, but the product's own reliability target requires 99.95%.
Quantifying the gap: a year has 8,760 hours. At 99.5% availability, permitted downtime is 0.5% of 8,760 = 43.8 hours per year. At 99.95%, permitted downtime is 0.05% of 8,760 = 4.38 hours per year. The vendor's contract therefore permits about 43.8 minus 4.38 = 39.42 hours per year more downtime than the product can actually tolerate.
Action: negotiated for a higher committed SLA where possible; where the vendor would not move the number, negotiated a compensation or credit clause tied to a downtime threshold, documented in writing. Regardless of the contract terms, added caching on the read path so a short vendor blip doesn't cascade immediately, and a fallback path that degrades the feature gracefully instead of erroring during an outage window.
Result: the contract negotiation raises the ceiling on paper, but the caching and fallback layer is what actually protects users during the gap between what the vendor promises and what the product needs, since a credit clause compensates you after an outage, it doesn't prevent one.

Trade-offs and pitfalls

• Mocks and stubs only help if kept in sync with the real contract. A stale mock creates a different kind of surprise at integration time.
• Vendor SLA credits are usually a small fraction of the real cost of downtime (lost trust, lost usage). Treat them as compensation, not as risk mitigation on their own, and pair them with technical fallbacks.
• Applying heavy contract-and-SLA process to a short, low-risk internal dependency slows down partners who need speed more than ceremony. Calibrate the rigor to the risk and duration of the dependency, not the same weight for every one.

Approach summary
Describe traffic, session model (sticky vs external), instance capacity (RPS, concurrent sessions, memory), target SLAs, and choose autoscaling signals (CPU, RPS per instance, queue length, custom metrics like active_sessions).

Capacity estimation formulas

• Estimate required instances by throughput:

instances_by_rps = ceil( peak_rps / instance_max_rps )

(instance_max_rps = measured median RPS an instance serves at target latency)

• Estimate by concurrent sessions (stateful / sticky):

instances_by_sessions = ceil( peak_concurrent_sessions / sessions_per_instance )

(sessions_per_instance = memory_limit_for_sessions / avg_session_size)

• Add headroom:

target_instances = ceil( max(instances_by_rps, instances_by_sessions) * (1 + headroom_fraction) )

(recommended headroom_fraction = 0.2–0.5 depending on burstiness)

Autoscaling thresholds & policies

• For external session store (stateless app servers): scale on RPS or request latency:
  • Scale out when RPS_per_instance > 0.7 * instance_max_rps or p95_latency > SLA
  • Scale in when RPS_per_instance < 0.4 * instance_max_rps for cooldown
• For sticky sessions (stateful): prefer scaling on queue length, new session rate, or manual/controlled scale to avoid rebalancing cost. Use a gentle scale rate and session draining.
• Example threshold formulas:

scale_out_when: (total_rps / current_instances) > 0.7 * instance_max_rps
scale_in_when:  (total_rps / current_instances) < 0.4 * instance_max_rps  AND current_instances > min_instances

Operational considerations

• Cooldown windows: 3–10 minutes; longer for sticky sessions.
• Draining: Evict new traffic, wait for session timeout or migrate sessions gracefully.
• Use health checks and circuit breakers; monitor session store latency (for external store).

Testing & validation

• Baseline profiling: measure instance_max_rps, avg_session_size, memory use, p50/p95 latency.
• Load tests: ramp to peak and 2x peak with realistic session patterns; validate scaling decision times, error rates, and session stickiness.
• Chaos/rolling tests: force instance termination to verify session draining and sticky behavior.
• Soak tests: sustained load for hours to capture memory leaks.
• Metrics to observe: p95 latency, error rate, CPU, memory, RPS_per_instance, session_store_latency, time_to_scale, dropped_sessions.

Trade-offs

• Sticky sessions simplify design but reduce flexibility and require careful drain logic; external session stores add complexity but enable faster autoscaling and better utilization. Choose based on SLA and operational readiness.

Overview
Below compares managed K8s (EKS/GKE/AKS) vs serverless containers (Fargate, Cloud Run) across responsibilities, use cases, cost, scaling, portability, and recommendations.

Operational responsibilities

• Managed K8s (EKS/GKE/AKS)
  • Cluster control plane managed; you manage node pools, CNI, ingress, RBAC, Helm, upgrades, autoscaling, monitoring, backup.
  • Responsible for pod scheduling, storage classes, CSI drivers, network policies.
• Serverless containers (Fargate / Cloud Run)
  • Provider manages infra, OS, scaling, patching. You supply container image and config (concurrency, memory, env).
  • Limited platform ops (observability, IAM, CI/CD).

Typical use cases

• Managed K8s: complex microservices, stateful apps (Databases, Kafka), custom networking, multi-workload clusters, infra requiring CRDs/operators.
• Serverless containers: stateless HTTP APIs, event-driven jobs, ephemeral workers, webhooks, simple background tasks.

Cost implications

• Managed K8s: pay for control plane (sometimes), and provisioned nodes (VMs) — may be inefficient at low utilization but good for high steady load via reserved/spot VMs.
• Serverless containers: pay-per-use (CPU/memory-per-second). Often cheaper for bursty/low-utilization workloads; can be expensive for constant high CPU/memory.

Scaling behavior

• Managed K8s: fine-grained control (HPA, VPA, Cluster Autoscaler). Can scale horizontally but node provisioning can add latency; can use node pools with mixed instance types and scale-to-zero is not native without addons.
• Serverless containers: rapid autoscale including scale-to-zero (Cloud Run) and near-instant concurrency scaling (Cloud Run supports multiple requests per instance). Fargate scales fast but often has higher cold-starts and per-task latency.

Vendor lock-in & portability

• Managed K8s: high portability (K8s API standard). Workload portability across clouds is strong; cluster configs, operators may reduce portability. More surface area but easier to move.
• Serverless containers: higher lock-in to platform behaviors (request timeouts, IAM, logging, scaling semantics). Container image portability exists, but operational model often ties you to provider.

Recommendations

• Bursty API services: prefer serverless containers (Cloud Run / Fargate) for rapid scale-to-zero, pay-per-use, simpler ops. Use Cloud Run for HTTP-heavy with concurrency needs; Fargate when deeper AWS integration or VPC-required.
• Long-running stateful workloads: prefer managed K8s to run statefulsets, operators, persistent volumes, and custom networking. Optimize cost with reserved instances/spot and node autoscaling.

Notes / Trade-offs

• Hybrid: run control-plane on K8s and front APIs on serverless for best of both worlds.
• Consider observability, security posture, and compliance when choosing.

Service selection
For AWS choose Amazon Aurora (MySQL/Postgres compatible) for high performance and serverless options; use RDS (Managed MySQL/Postgres) if cost/compatibility matters. On GCP use Cloud SQL; on Azure use Azure Database for MySQL/Postgres. Choose the provider that matches existing stack, SLA and read/write patterns.

Sizing

• Pick instance class based on CPU/memory needs from load testing (e.g., db.m6i.large → scale up).
• Start with balanced vCPU/memory; prefer vertical scale-friendly families.
• Storage: use gp3 (AWS) or SSD provisioned IOPS for predictable IO; size with growth buffer and autoscaling if supported.

Backups & retention

• Enable automated daily snapshots; retention 7–35 days per compliance.
• Configure point-in-time recovery with binary/log shipping; test restores regularly (quarterly).

High availability & automated failover

• Use Multi-AZ / regional primary-replica configuration. For Aurora, use cluster endpoints and automatic failover <30s. Enable synchronous replication where possible.

Read scaling

• Add read replicas (Aurora readers or RDS read-replicas). Use load balancer or application-aware routing to distribute reads. Monitor replica lag and promote if needed.

Encryption

• Enable at-rest encryption using cloud KMS customer-managed keys. Enforce TLS for in-transit. Rotate keys per policy and restrict KMS IAM roles.

Maintenance windows & routine practices

• Set weekly maintenance window during low traffic. Apply minor patches regularly, major upgrades in planned maintenance with blue/green or snapshot rollback plan.
• Routine: monitor metrics (CPU, IOPS, connections, replica lag), run failure drills, automate backups validation, review slow query logs and index optimization, cost review and right-sizing monthly.

This approach balances reliability, performance, security and operational readiness for production workloads.

Direct answer

In forty-eight hours I am not trying to understand the technology, I am trying to deliver one narrow, correctly-working slice of it and be honest about everything I did not verify. I spend the first couple of hours scoping exactly what "real" has to mean for the deliverable, deliberately decide what to fake, stub, or hard-code outside that slice, and I protect the work by verifying the riskiest part by hand rather than trusting untested intuition, then naming the residual risk explicitly to whoever receives the work.

Structured elaboration

• Scope ruthlessly from the actual deliverable backward: what is the smallest real thing that satisfies the ask, and what can be stubbed, mocked, hard-coded, or simply omitted for now.
• Name out loud what is being skipped and why: edge cases, error handling for paths not exercised, configuration options, anything the tool offers that this specific window does not need.
• For the part that has to be real, verify by hand what you cannot yet trust your own understanding to catch: manually walk a request through, check a response against documentation line by line, rather than relying on "it looked right" for the piece that matters most.
• Where existing knowledge partly maps from something familiar, be explicit with yourself about which parts of that intuition are actually being verified and which are just being trusted, since a partial map is exactly where false confidence creeps in.
• Flag residual risk explicitly to whoever receives the work: what was not verified, what could break outside the narrow case tested, and what should be checked next if this needs to become durable.

Worked example

With about forty-eight hours' notice, I was asked to integrate a third-party payment provider's webhook into a live service for a stakeholder demo the next day, having never touched that provider's interface before. I scoped the real slice tightly: handle exactly one webhook event type correctly, with real signature verification, since faking that would be dangerous even in a demo, and hard-coded a canned response for every other event type in the provider's catalog rather than trying to handle all of them. I verified the signature-verification code by hand against the provider's documented example payload and hash, byte by byte, rather than trusting that it compiled and ran without error, since that was exactly the part I could not yet trust my own instincts on. I left retry and duplicate-delivery handling explicitly out of scope, wrote that down in the change description, and told the person receiving the work directly that a duplicate webhook delivery would currently be processed twice, so it was not safe to treat as production-ready before that gap closed.

Trade-offs and pitfalls

• The biggest failure mode under this kind of compression is quietly treating "it ran once without an error" as proof of correctness; hand-verifying the riskiest slice is exactly what prevents that.
• Skipping too aggressively can produce a demo that looks complete and creates false confidence that the hard part is done, when the hard part was actually the part left out; naming what was skipped, out loud, is what prevents that.
• Leaning on knowledge that only partly maps from a familiar tool is efficient but dangerous if the transferable parts are not separated from the parts that merely look similar.

Direct answer

Node labels, taints/tolerations, affinity, and topology spread each answer a different scheduling question, and this scenario needs all of them combined rather than one chosen over the others. Taints on the volatile pool (spot) provide the exclusion guarantee that keeps critical workloads off it by default. Affinity provides the positive pull that sends the right workload to the right pool, and for GPU (graphics processing unit) nodes that pull has to be paired with the device-plugin's extended resource accounting, not just a label. Topology spread (or pod anti-affinity) spreads critical replicas so a single node or zone loss never removes more than one replica.

Structured elaboration

GPU nodes need more than a label

Labeling a node hardware=gpu is necessary but not sufficient. The scheduler only knows a node has usable GPU capacity once the NVIDIA device plugin (or a vendor equivalent) DaemonSet advertises it as an extended resource in the node's allocatable, for example nvidia.com/gpu: 4. A pod requests it like any other resource:

resources:
  limits:
    nvidia.com/gpu: 1

Without the resource request, a pod with only nodeSelector: {hardware: gpu} can land on a GPU node without ever actually reserving a device, or worse, several pods could expect the same already-claimed device. Pair the label or affinity rule (which steers the pod to the right node family) with the extended-resource request (which makes the scheduler actually reserve a device).

Excluding spot from critical workloads

kubectl taint nodes -l node-lifecycle=spot spot=true:NoSchedule

Only pods that explicitly tolerate it can land there:

tolerations:
- key: "spot"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Critical services carry no such toleration, so the scheduler treats spot nodes as invisible to them by default, with no risk of a nodeSelector typo accidentally placing one there.

Spreading critical replicas

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-service

whenUnsatisfiable: DoNotSchedule makes this a hard constraint; ScheduleAnyway makes it best-effort. For a service that must survive a zone loss, DoNotSchedule is the right choice even though it can leave a pod Pending if the under-represented zone runs out of capacity, which is a signal to add capacity, not to loosen the constraint.

Priority for the pool that can vanish without warning

Spot capacity can disappear with only seconds of notice. Give critical workloads a higher priorityClassName so, if critical and batch pods ever land in the same resource pool during a capacity burst, the scheduler preempts lower-priority batch pods rather than the reverse.

Worked example

A cluster has 2 GPU nodes (4 GPUs each, 8 total), 3 on-demand nodes for critical services, and 4 spot nodes for batch. A training job requests 2 GPUs; with the device plugin installed, allocatable nvidia.com/gpu across the 2 GPU nodes totals 8, and the scheduler places the pod on whichever GPU node currently has 2 or more free, refusing to schedule (Pending, correctly) if all 8 are already claimed elsewhere. A critical service asks for 3 replicas with topologySpreadConstraints maxSkew: 1 across the 3 on-demand zones: with exactly 3 zones and 3 replicas the constraint is satisfiable exactly, one per zone, so losing any single zone loses exactly 1 of 3 replicas, never more.

Trade-offs and pitfalls

• A taint on the spot pool only stops pods without the toleration; it does not, by itself, stop a batch pod from landing on an on-demand node. If batch must also be kept off on-demand capacity, that pool needs its own taint plus a matching toleration on batch pods, not just a one-directional rule.
• Required affinity and required topology spread can leave pods unschedulable when capacity is tight. Monitor scheduling events (Pending plus FailedScheduling) rather than reflexively loosening a hard rule to preferred, which quietly reintroduces the correlated-failure risk the rule existed to prevent.
• GPU device counts are not overcommittable the way CPU or memory can be with limits set above requests. Plan capacity on whole-device granularity.

Direct answer. Serve the API from multiple regions behind Route 53 latency based or geoproximity routing, terminate each region behind a Network Load Balancer (NLB, for raw Layer 4 throughput and client IP preservation) or Application Load Balancer (ALB, for Layer 7 routing and AWS Web Application Firewall integration), and fan out to an autoscaled fleet in private subnets across at least three Availability Zones (AZs). At 10,000+ transactions per second the design itself is the easy part; the soft quotas on Elastic Network Interfaces (ENIs, the virtual network cards attached to instances, containers, and load balancer nodes), ALB target registration, and Route 53 API and record counts are what actually break first, and each has a known architectural workaround rather than just a support ticket.

Reference topology

flowchart TB
    DNS[Route 53\nlatency-based routing] --> R1[Region A entry]
    DNS --> R2[Region B entry]
    subgraph R1[Region A]
        NLB1[NLB / ALB] --> ASG1[Autoscaling group\nAZ-a, AZ-b, AZ-c]
        ASG1 --> VPCE1[VPC endpoints\nS3, DynamoDB, Secrets Manager]
        ASG1 --> NAT1[NAT gateway per AZ]
    end
    subgraph R2[Region B]
        NLB2[NLB / ALB] --> ASG2[Autoscaling group\nAZ-a, AZ-b, AZ-c]
    end
    CF[CloudFront edge cache] --> DNS
    Shield[AWS Shield + WAF] --> CF

Where the soft quotas actually bite

NLB vs ALB at this scale

A common hybrid for this scale is an NLB in front of one or more ALBs: the NLB absorbs connection volume and gives a stable set of IPs for Route 53 or on-premises peers, while the ALBs behind it do the L7 routing and WAF enforcement, so no single ALB has to hold every target.

Design targets if this had to scale toward 1,000,000 requests per second

The same soft-quota discipline extends past 10,000 TPS. At the 1,000,000 requests per second design point, three additional bottleneck classes move from "watch" to "must design around":

• Connection handling. At that volume the number of concurrent TCP connections, not just requests, becomes the binding constraint: enable NLB cross zone load balancing and keep-alive reuse on the client side to avoid a connection-per-request pattern, and size target group connection draining and idle timeouts so scale-in events do not cause a thundering herd of reconnects.
• Caching. Push everything cacheable to the edge before it reaches the load balancer at all. CloudFront in front of Route 53's origin, plus ElastiCache for computed responses that are expensive to regenerate, is what turns a 1,000,000 requests per second edge number into a much smaller number of requests that actually reach the autoscaling group, which is the only way the ENI and target quotas above stay survivable.
• DDoS (Distributed Denial of Service) protection. Volumetric floods are a distinct failure mode from organic growth. AWS Shield Advanced plus WAF rate based rules in front of CloudFront absorb the attack traffic at the edge so it never consumes ALB capacity units, NAT (Network Address Translation) gateway bandwidth, or backend autoscaling capacity that legitimate traffic needs.

Trade-offs and pitfalls

• NAT gateway placement. One NAT gateway per AZ (not one shared NAT for the Region) is required for AZ independence; a single shared NAT recreates a cross AZ single point of failure and adds cross AZ data transfer cost on every outbound call.
• VPC endpoints reduce NAT load, but only for AWS services. Gateway endpoints exist only for S3 and DynamoDB; every other AWS service (Secrets Manager, ECR, Systems Manager) needs an Interface endpoint (a PrivateLink ENI in each subnet) to get off NAT gateway bandwidth, which itself consumes ENIs and adds to the same soft quota this design is trying to protect.
• Requesting increases late. ENI, target, and record quota increases are not instant; requesting them the week of a launch instead of during capacity planning is the single most common way this design fails in production despite being architecturally correct on paper.
• Treating TLS termination point as a style choice. Where TLS terminates changes who can see plaintext, what WAF can inspect, and how much CPU the load balancer layer spends versus the backend; it is a security and cost decision, not just a configuration toggle.

Direct answer

A scalable, multi-account, multi-cluster GitOps pipeline needs its ROLLBACK PROCEDURE for a FAILED reconciliation to be a first-class design element, not an afterthought, because at this scale a partial reconciliation failure (some manifests applied, others rejected mid-sync) is a routine operational event, not an edge case. The design combines a repo layout mirroring the account/cluster/environment hierarchy, a controller instance per cluster (never one controller reaching across account boundaries, which would undermine the RBAC isolation the requirements explicitly call for), and an explicit, tested procedure for what happens when a sync fails PARTWAY, distinct from ordinary drift-correction rollback.

Structured elaboration

Repository layout. A structure mirroring the real topology: clusters/<cloud-account>/<cluster-name>/<environment>/, with shared platform manifests factored into a common base that each cluster-environment combination overlays. This makes "which cluster does this manifest actually apply to" answerable by directory path alone, essential once the fleet spans enough accounts and clusters that tribal knowledge stops scaling.

Automated reconciliation with drift detection. One controller instance PER CLUSTER (not one central instance spanning every account), each watching only its own directory in the shared repo, so a compromise or misconfiguration in one cluster's controller cannot reach across the account boundary to another.

Role-based access controls per environment. Each environment's controller service account holds Kubernetes RBAC scoped to exactly that cluster; human access to trigger manual syncs or view status is scoped via the GitOps tool's own project/tenancy construct, matching the SAME account/cluster/environment boundaries the repo layout encodes, so the repo structure, the RBAC structure, and the controller topology all reflect the SAME hierarchy rather than three independently-maintained mappings that can drift out of sync with each other.

Approval gates for production. A required-review gate on any PR touching a prod environment directory, structurally enforced via CODEOWNERS and branch protection, distinct from and in addition to non-production environments' lighter review bar.

Secure handling of secrets. Per-cluster External Secrets Operator instances, each scoped to that cluster's own path prefix in the secrets backend, so a promotion between environments never requires touching a secret VALUE directly, only the reference structure.

Rollback procedures for failed reconciliations, the part this question specifically emphasizes. Distinguish two failure shapes: (1) a sync REJECTED entirely before applying anything (a validation or policy-as-code failure catches it up front), which needs no rollback at all, nothing was ever applied; and (2) a sync that applied SOME resources before failing partway (a dependency ordering issue, a transient API error on one specific resource), which leaves the cluster in a genuinely INCONSISTENT intermediate state, some resources reflecting the new desired state, others still on the old one. For case 2, the controller's own automatic retry (most GitOps controllers retry a failed sync automatically on the next reconciliation cycle) is usually sufficient IF the underlying cause was transient; if it is not transient (a genuine configuration error), the safe procedure is reverting the OFFENDING commit specifically (not the whole recent history) and letting reconciliation catch the cluster back up to the last known-good state, the same Git-revert-and-reconcile pattern used for ordinary rollback, but explicitly TESTED against a partial-failure scenario before relying on it in production, since a partial failure's exact intermediate state is harder to reason about than a clean, fully-applied-then-reverted change.

Worked example

A concrete PR/merge workflow tying the pieces together: a change merges into clusters/account-b/cluster-3/prod/, requiring @org/platform-team approval (CODEOWNERS) before merge is even possible; on merge, cluster-3's dedicated controller instance (holding RBAC scoped only to that cluster) picks up the change and begins syncing; if the sync fails after applying 6 of 9 manifests (a dependency ordering problem causes the 7th to fail), the controller's automatic retry attempts the remaining 3 on the next cycle; if the underlying issue is NOT transient (a genuine error in the 7th manifest), an operator reverts specifically that commit, and reconciliation, on its next cycle, reconciles the cluster back to matching the reverted (last known-good) declared state, resolving the partial-application inconsistency without requiring a manual, resource-by-resource cleanup.

Trade-offs and pitfalls

• Common mistake: designing rollback procedures only around "a bad change reached prod and needs reverting" and never explicitly testing the PARTIAL-failure case, a sync that applies 6 of 9 manifests before failing leaves a genuinely different, harder-to-reason-about intermediate state than a clean, fully-applied bad change; if this specific case has never been deliberately exercised (a game-day exercise, or a staging-environment fault injection), the team's confidence in "we can just revert and reconcile" may not survive contact with a real partial failure.
• One controller instance per cluster, rather than one centralized instance across accounts, is the right default here specifically because the requirements name per-tenant RBAC and account isolation explicitly; a centralized controller would need its OWN cross-account credentials to reach every cluster, which is exactly the concentration-of-risk a single shared credential creates for isolation-sensitive requirements.
• The repo-layout, RBAC-structure, and controller-topology hierarchies all mirroring the SAME account/cluster/environment structure is a deliberate design choice, not a coincidence, letting any one of the three drift out of alignment with the others (a repo reorganization that doesn't get reflected in RBAC scoping, for instance) reintroduces exactly the kind of tribal-knowledge dependency this design exists to eliminate at scale.
• Automatic retry on transient failures is genuinely helpful but can mask a persistent problem if not paired with alerting on REPEATED failures for the same resource, a sync that keeps failing and retrying silently, cycle after cycle, without ever escalating to a human, is a worse outcome than a sync that fails once, loudly, and gets a deliberate revert.

Direct answer: Structure the cutover as: pre-migration validation, then continuous change-data-capture (CDC) replication to build and maintain a synchronized target, then a short cutover window (stop writes, drain lag, verify, repoint), with explicit, pre-agreed rollback triggers rather than an ad hoc judgment call during the window.

Structured elaboration. Pre-migration validation: confirm the target database is provisioned and tuned correctly (indexes, connection limits, parameter groups matching or exceeding source), run a full initial data load and a first-pass checksum/row-count comparison BEFORE relying on it for anything, and validate the CDC pipeline itself on non-critical tables first. CDC or replication setup: enable log-based replication from source to target, let it run long enough to validate it keeps up under real production write volume (not just during a quiet testing window), monitor replication lag continuously. Synchronization verification: periodic (e.g., hourly) row-count and checksum comparisons on a sample of tables throughout the replication period, not just once at the end, so a silently-diverging replication stream is caught early rather than discovered at cutover. Final cutover sequence: (1) announce the maintenance window to stakeholders, (2) put the application into a brief write-pause or read-only mode, (3) wait for replication lag to reach zero, (4) run a final full parity check (row counts and checksums across all migrated tables, not a sample), (5) repoint the application's connection string to the target, (6) remove the write-pause. Testing/verification after cutover: run the application's smoke tests against the new database, monitor error rates and query latency for an elevated period post-cutover, and keep the source database available (read-only) for a defined bake period in case rollback is needed. Explicit rollback triggers: define BEFORE the cutover window, not during it: e.g., "if the final parity check finds any discrepancy," "if replication lag doesn't reach zero within 15 minutes of write-pause," or "if post-cutover error rate exceeds 2x baseline within the first 30 minutes" all trigger an immediate rollback to the source, no live debate required in the moment.

Worked example. For a database expected to have a 10-minute maintenance window: minutes 0-5 write-pause and lag-drain (the phase does not advance until lag reaches zero; if lag has NOT reached zero by minute 5, that is itself the rollback trigger, and the plan aborts here rather than proceeding into parity checking against a target that never finished catching up), minutes 5-8 final parity check across all tables (only entered once lag-drain succeeded within its 5-minute budget), minutes 8-9 repoint application and run smoke tests, minutes 9-10 remove write-pause and begin elevated post-cutover monitoring. If the rollback trigger fires at minute 5, the in-progress parity-check phase never starts: writes resume against the still-authoritative source immediately, and the target is discarded rather than repaired, since nothing has been repointed to it yet. Tools: a managed CDC/migration service for replication, a lightweight parity-check script (checksums per table, compared source vs. target) run both periodically during replication and as the final gate.

Trade-offs & pitfalls. Defining rollback triggers DURING the cutover window under time pressure, rather than agreeing them beforehand, is the most common way a fixable problem turns into an extended, improvised outage: a pre-agreed "if X, then rollback, no discussion" rule removes the temptation to push through a warning sign because the team is already committed to the window.

Direct Answer

Route queries by data age: hot is optimized for write and point-lookup latency at full resolution, warm keeps full resolution but trades write-path speed for better compression, and cold and archive downsample aggressively and lean on cheap object storage, accepting slower, block-oriented reads. The tiering only pays off if you also downsample, keeping full resolution for 7 years costs an order of magnitude more than the numbers below show.

Structured Elaboration

Rehydration is bounded by making every tier block-addressable by time range and label hash: a query for a specific series over a specific window only pulls the blocks that could contain it, using the manifest's Bloom filters (compact per-block structures that can say "this block definitely doesn't contain that series" without opening it) to skip the rest, rather than scanning a tier wholesale.

Tier flow

flowchart LR
  Q["Query Router"] --> HOT[("Hot: 1 day")]
  Q --> WARM[("Warm: 30 days")]
  Q --> COLD[("Cold: 1 year, downsampled")]
  Q --> ARC[("Archive: 7 years, downsampled")]
  HOT -->|"compaction"| WARM
  WARM -->|"downsample to 5min"| COLD
  COLD -->|"downsample to 1hr"| ARC
  ARC -.->|"on-demand rehydrate"| Q

Worked Example

Assume 1,000,000 active series, each scraped every 15 seconds (a stated design input). Raw points per series per day:

Total raw points/day across the fleet: $1{,}000{,}000 \times 5{,}760 = 5.76\times10^{9}$.

Hot (1 day). Assume time-series-optimized compression achieves roughly 2 bytes/point (a stated assumption for delta-of-delta timestamp and XOR value encoding, not a cited benchmark), plus 30% index overhead:

Warm (30 days). Larger compaction blocks improve the ratio slightly to 1.5 bytes/point, still at full resolution:

Cold (1 year). Downsample to 5-minute resolution, a 300s/15s = 20x reduction in point count, at 2 bytes/point:

Had the cold tier kept raw resolution instead, the year would cost $5.76\times10^{9} \times 365 \times 2\ \text{B} \approx 4{,}204.8\ \text{GB}$, about 20x more, which is the whole reason the downsample step exists.

Archive (years 2 to 7, 6 years). Downsample further to 1-hour resolution (24 samples/day), at 1.5 bytes/point since bulk cold storage compresses better:

Total 7-year footprint (before replication): $15.0 + 259.2 + 210.2 + 78.8 \approx 563.2\ \text{GB}$. Kept at raw resolution for all 7 years instead, the footprint would be roughly $4{,}204.8 \times 7 \approx 29{,}434\ \text{GB} \approx 29.4\ \text{TB}$, about 52x larger. That gap is the entire economic argument for tiering with downsampling rather than tiering on storage class alone.

Trade-offs and Pitfalls

Downsampling is lossy by design: once a cold-tier block collapses 20 raw points into one 5-minute aggregate, you cannot recover the original spikes inside that window. Decide per metric type whether that is acceptable (aggregate SLI counters, usually fine) or not (a metric feeding an anomaly-detection model that needs the raw shape, usually not, and should either stay hot longer or get its own retention policy).

Cross-tier query stitching is a common failure point: a dashboard spanning "last 45 days" crosses the hot/warm boundary and the warm/cold boundary is close by too, and the query planner has to merge full-resolution warm data with 5-minute cold data without a visible discontinuity in the chart. Test this seam explicitly, it is where tiering bugs show up in production, not inside a single tier.

Archive rehydration cost is easy to underestimate: a 1-hour restore-initiation SLA sounds fast until a customer asks for a 3-year lookback across 1,000 series and that pulls thousands of archive blocks, each carrying its own restore latency and egress cost. Expose rehydration as an explicit, rate-limited, customer-visible operation rather than pretending archive queries behave like cold-tier queries.