Cloud Migration Strategy and Execution Questions

Direct answer: A rollback after a failed cutover has to account for the DELTA of writes that landed on the new system before the failure was caught, not just "restore the old system and repoint": failing to reconcile that delta is the most common way a rollback itself causes data loss.

Structured elaboration. Delta between cutover attempt and rollback: from the moment traffic was repointed to the new database until the rollback decision, any writes that succeeded against the new system exist ONLY there; simply repointing back to the old (now stale) system silently discards them unless they're captured and reconciled. Ongoing writes during rollback: if the rollback itself takes any non-zero time, the application needs to either be paused (safest, but extends the outage) or explicitly handle writes landing on a system that's mid-rollback, which is risky enough that pausing writes is usually the right trade despite the extra downtime. Necessary restoration steps: (1) pause writes immediately upon deciding to roll back, (2) extract every write that landed on the new (failed) system since cutover (from the new system's transaction log or change-data-capture (CDC) stream, not from application logs, which are an unreliable secondary source), (3) replay those writes onto the old system in the correct order, handling any conflicts (e.g., a row updated on both systems independently, which shouldn't happen if writes were properly paused during cutover but needs a defined resolution rule if it does), (4) verify the old system now reflects every write that was ever accepted anywhere, (5) repoint the application back to the old system, (6) resume writes. Communications to users: acknowledge the issue and expected resolution time as soon as the rollback decision is made, not after it completes, since a longer-than-usual outage without communication erodes trust faster than the outage itself; a follow-up communication after resolution should be honest about whether any data or functionality was affected. Tests to validate rollback succeeded: row-count and checksum parity between what the new system had at the moment of failure and what the old system has post-replay (confirming no writes were lost in the replay), plus application-level smoke tests against the old system to confirm normal operation resumed cleanly.

Worked example. Say cutover happened at 2:00 PM, and a critical bug was discovered at 2:20 PM. Every write accepted by the new database between 2:00 and 2:20 (order IDs, inventory updates, etc.) must be extracted from the new system's transaction log and replayed onto the old system before repointing back, in the order they originally committed, so that the 20 minutes of activity isn't simply lost. If replay can't complete cleanly (e.g., a write on the new system conflicts with a change that had ALSO happened on the old system, which would indicate the write-pause during original cutover didn't fully take effect), that's escalated as a data-integrity incident, not resolved by an automated best-guess.

Trade-offs & pitfalls. The most damaging shortcut is announcing a rollback is complete once traffic is repointed, without confirming the delta-replay actually succeeded and was verified: a rollback that silently drops 20 minutes of orders is arguably worse than the original problem that triggered it, since it converts a visible outage into an invisible data-loss incident.

Direct answer: A 25% post-migration latency increase needs a structured, elimination-based troubleshooting approach: first confirm the regression is real and consistent (not a measurement artifact), then isolate WHERE in the request path the added latency is occurring, then determine whether the cause is genuinely cloud-specific (network topology, a different managed-service's performance characteristics) versus a migration-unrelated coincidence.

Structured elaboration. What telemetry to collect: distributed tracing spans covering the full request path (to see which specific hop or component accounts for the added latency, rather than just an aggregate end-to-end number), infrastructure-level metrics for the new environment (CPU/memory/network saturation, since a resource constraint that didn't exist on-prem could be the direct cause), and a side-by-side comparison against the PRE-migration baseline for the same traffic pattern (confirming this is genuinely a regression and not, say, a seasonal traffic increase that happened to coincide with the migration). How to isolate new cloud-specific factors: compare latency broken down BY COMPONENT (is the added latency in the application tier, the database, a network hop, a managed service that replaced a self-hosted one) rather than treating it as one undifferentiated number; cloud-specific candidates to check specifically include cross-availability-zone network hops that didn't exist in the on-prem topology, a managed database service's different performance characteristics under the same query patterns (e.g., a managed service may have different connection-pooling or cold-start behavior), and DNS/service-discovery resolution differences. Short-term mitigations to reduce user impact: if a specific bottleneck is identified but not yet fully understood/fixed, targeted mitigations might include caching to reduce load on a slow component, adjusting connection-pool or timeout settings that may not have been re-tuned for the new environment's characteristics, or, if the increase is severe enough, a partial rollback of the specific affected component while the root cause is investigated more thoroughly, rather than accepting the regression indefinitely while investigation continues.

Worked example. Distributed tracing reveals the added latency is concentrated in database query time, not in application logic or network transit. Comparing the new managed database's configuration against the old self-hosted one finds the connection pool size was left at a default that's smaller than what the on-prem setup had been tuned to, causing connection queueing under the same traffic load. Short-term mitigation: increase the connection pool size to match the previously-tuned on-prem value, immediately reducing the queueing-driven latency; longer-term, re-run the same performance-tuning exercise the on-prem database originally went through, since a straight migration doesn't automatically carry forward tuning that was done empirically over time on the old system.

Trade-offs & pitfalls. Assuming the regression must be "just how the cloud is" (accepting a performance hit as an inherent cost of migration) without actually tracing to a specific, addressable cause is a common way a fixable configuration gap (like the untuned connection pool above) goes unaddressed indefinitely; most post-migration latency regressions have a specific, findable cause, not a vague "cloud is different" explanation.

Direct answer: Structure the cutover as: pre-migration validation, then continuous change-data-capture (CDC) replication to build and maintain a synchronized target, then a short cutover window (stop writes, drain lag, verify, repoint), with explicit, pre-agreed rollback triggers rather than an ad hoc judgment call during the window.

Structured elaboration. Pre-migration validation: confirm the target database is provisioned and tuned correctly (indexes, connection limits, parameter groups matching or exceeding source), run a full initial data load and a first-pass checksum/row-count comparison BEFORE relying on it for anything, and validate the CDC pipeline itself on non-critical tables first. CDC or replication setup: enable log-based replication from source to target, let it run long enough to validate it keeps up under real production write volume (not just during a quiet testing window), monitor replication lag continuously. Synchronization verification: periodic (e.g., hourly) row-count and checksum comparisons on a sample of tables throughout the replication period, not just once at the end, so a silently-diverging replication stream is caught early rather than discovered at cutover. Final cutover sequence: (1) announce the maintenance window to stakeholders, (2) put the application into a brief write-pause or read-only mode, (3) wait for replication lag to reach zero, (4) run a final full parity check (row counts and checksums across all migrated tables, not a sample), (5) repoint the application's connection string to the target, (6) remove the write-pause. Testing/verification after cutover: run the application's smoke tests against the new database, monitor error rates and query latency for an elevated period post-cutover, and keep the source database available (read-only) for a defined bake period in case rollback is needed. Explicit rollback triggers: define BEFORE the cutover window, not during it: e.g., "if the final parity check finds any discrepancy," "if replication lag doesn't reach zero within 15 minutes of write-pause," or "if post-cutover error rate exceeds 2x baseline within the first 30 minutes" all trigger an immediate rollback to the source, no live debate required in the moment.

Worked example. For a database expected to have a 10-minute maintenance window: minutes 0-5 write-pause and lag-drain (the phase does not advance until lag reaches zero; if lag has NOT reached zero by minute 5, that is itself the rollback trigger, and the plan aborts here rather than proceeding into parity checking against a target that never finished catching up), minutes 5-8 final parity check across all tables (only entered once lag-drain succeeded within its 5-minute budget), minutes 8-9 repoint application and run smoke tests, minutes 9-10 remove write-pause and begin elevated post-cutover monitoring. If the rollback trigger fires at minute 5, the in-progress parity-check phase never starts: writes resume against the still-authoritative source immediately, and the target is discarded rather than repaired, since nothing has been repointed to it yet. Tools: a managed CDC/migration service for replication, a lightweight parity-check script (checksums per table, compared source vs. target) run both periodically during replication and as the final gate.

Trade-offs & pitfalls. Defining rollback triggers DURING the cutover window under time pressure, rather than agreeing them beforehand, is the most common way a fixable problem turns into an extended, improvised outage: a pre-agreed "if X, then rollback, no discussion" rule removes the temptation to push through a warning sign because the team is already committed to the window.

Direct answer: An observability plan for a migration should track system metrics (latency, error rate, saturation), business metrics (the actual outcomes the migration shouldn't disrupt: conversion rate, transaction volume), logs, and distributed traces spanning the cutover boundary, with alert thresholds calibrated against the PRE-migration baseline specifically, not generic defaults, since "normal" for this system is whatever it was before, not an industry-standard number.

Structured elaboration. Metrics (system and business): system metrics are the standard SRE set (request latency percentiles, error rate, resource saturation) but business metrics matter equally for a migration: if latency and error rate look fine but conversion rate or transaction success rate drops, that's a real regression the system metrics alone would miss (e.g., a subtle correctness bug that doesn't throw errors but produces wrong results). Log and tracing spans: ensure logging and tracing coverage spans the actual cutover moment and the immediate stabilization window with enough granularity to reconstruct what happened if an issue surfaces, and specifically tag/label telemetry from the migration window so it can be easily filtered and compared against pre/post baselines later. Alert thresholds: set relative to the PRE-migration baseline (e.g., "alert if p99 latency exceeds baseline by 50%") rather than an absolute number that might not reflect this specific system's normal behavior; this also means capturing a clean baseline BEFORE migration starts, not estimating one after the fact. Dashboards: build a migration-specific dashboard showing old-vs-new comparison side by side for the key metrics, used actively during the cutover window and the stabilization period, not assembled after the fact for a retrospective. SLOs/KPIs to define and monitor: the existing SLOs (if any) should apply unchanged to the new environment (the migration shouldn't be an excuse to relax a commitment), plus migration-specific KPIs (e.g., "percentage of requests served without an error attributable to migration" as an explicit, temporary tracked metric during the transition). During a migration wave and the immediate post-migration stabilization period: monitoring intensity should be highest immediately around cutover (tight windows, low alert-threshold tolerance, active human attention) and can relax to normal operational monitoring once the system has demonstrated stability over an agreed stabilization period, rather than staying in a permanently heightened, unsustainable alert posture.

Worked example. A migration-specific dashboard shows, side by side: pre-migration baseline p50/p95/p99 latency versus live post-cutover values, pre- vs post-migration error rate, and a business metric (e.g., checkout completion rate) tracked continuously through the cutover window; alerts are configured at baseline+30% for latency and baseline+2x for error rate specifically for the 48-hour stabilization window following cutover, then revert to the system's normal (non-migration-specific) alerting configuration once stability is confirmed.

Trade-offs & pitfalls. Relying only on system-level metrics (latency, error rate) without a business-outcome metric is a common gap: a migration can introduce a subtle correctness issue (a currency-conversion bug, a mis-mapped field) that produces no errors and no latency change at all, and only shows up as an unexplained dip in a business metric like completed transactions.

Direct answer: When replication lag is climbing beyond the acceptable RPO (recovery point objective) during a live cutover, the immediate priority is stopping the lag from getting worse (identify and address the bottleneck if it's fixable in real time) while communicating a clear, honest status; the decision to pause, roll back, or proceed hinges on whether the lag trend can be reversed within the remaining cutover window and whether proceeding anyway would violate the actual RPO commitment.

Structured elaboration. Immediate mitigation steps: first, diagnose WHY lag is increasing (a spike in write volume on the source, a resource constraint on the replication target, a network issue on the replication path) since the right mitigation differs by cause; a resource constraint might be addressable in real time (scale up the target's replication-apply capacity), while a sustained write-volume spike on the source might not be something you can fix mid-cutover at all. Communications to stakeholders: an honest, specific status update ("replication lag is currently at X, trending [up/stabilizing], RPO commitment is Y, decision point is in Z minutes") rather than either false reassurance or an alarming update with no actionable information; stakeholders need enough detail to understand the actual risk, not just "we're watching it." Decision logic (pause, roll back, or proceed): PAUSE (hold the cutover at its current stage, neither completing nor reverting) is appropriate if lag is climbing but the root cause looks addressable within a reasonable window and reverting isn't yet necessary; this buys diagnosis time without committing either direction. ROLL BACK is appropriate if the root cause isn't fixable within the time available before the RPO commitment would be breached, or if lag is climbing with no clear cause (an unexplained trend is itself a red flag, since proceeding on data you can't currently explain the state of is risky for anything with real integrity requirements). PROCEED is appropriate ONLY if lag stabilizes and drains back within an acceptable range before the actual cutover moment, confirmed by evidence (the lag metric trending back down, not just "hopefully it'll be fine"), never based on hope that it'll resolve itself under continued pressure.

Worked example. Lag climbs from a normal few seconds to 90 seconds against an RPO commitment of 60 seconds, with 20 minutes remaining before the scheduled cutover window closes. Diagnosis finds the replication target is CPU-constrained (from concurrent unrelated batch jobs sharing the instance). Mitigation: pause the competing batch jobs, freeing capacity; lag begins trending back down within 5 minutes. Decision: PAUSE the cutover (don't proceed yet, don't roll back either) while monitoring the recovery trend; once lag is confirmed back under the 60-second RPO threshold and holding for a stable period (not just a momentary dip), proceed with the cutover. If instead the diagnosis had found an unexplained, non-recoverable source-side issue with no clear fix available in the remaining window, the correct call would be ROLLBACK, even though that means missing the scheduled cutover window, because proceeding with lag exceeding the RPO commitment on a system where that commitment matters is a worse outcome than a delayed cutover.

Trade-offs & pitfalls. The most damaging failure mode here is proceeding with the cutover HOPING lag will resolve itself, without diagnosing a specific, addressable cause; that's a bet, not a decision, and it's precisely the kind of judgment call a pre-agreed decision framework (diagnose first, then pause/rollback/proceed based on evidence, not hope) is meant to prevent under time pressure.