Safe Deployment and Rollback Strategies Questions

Direct answer
A 1.5x latency increase with a flat error rate during a canary is exactly the ambiguous case automated canary analysis is built for: it's not an obvious failure (nothing's erroring) but it's also not obviously fine, so the right move is a structured diagnostic pass, not an immediate gut call either direction.

Structured elaboration
Step-by-step, in order:

1. Confirm it's real, not a sample-size artifact: check the canary's request count for this window; 5% traffic might be a small enough sample that a couple of slow requests skew the p95/p99 without it being a genuine, broad regression.
2. Check WHICH percentile moved: did the mean move, or specifically the tail (p99)? A tail-only shift suggests a subset of requests hitting a slow path (a cold cache, a specific input shape), while a broad shift across all percentiles suggests something more systemic.
3. Compare against infrastructure-level signals: CPU/memory on the canary pods specifically, are they resource-constrained relative to stable? A canary running on fewer instances than the stable fleet can look "slower" purely from having less capacity per request, not from a code regression.
4. Trace a slow request: pull a distributed trace for one of the slow requests and see WHERE the extra time is going; is it in the new code path itself, or in a downstream dependency call that both versions share (which would point away from the deploy as the cause)?
5. Check logs for the canary specifically: any new warnings, retries, or timeout patterns that correlate with the deploy?
6. Decide: continue if the increase traces to a benign, expected cause (e.g., cold cache that's now warming) and the trend is improving; pause and gather more data if the cause isn't yet clear and you have time to wait; roll back if the trace points to a genuine regression in the new code or if the trend is worsening rather than stabilizing.

Worked example
Tracing a slow canary request shows the extra ~150ms is spent in a downstream inventory-service call that BOTH old and new code make identically, ruling out the new code as the cause; checking resource metrics shows the canary pods are running at higher CPU utilization simply because the canary slice has fewer replicas than its 5% traffic share would proportionally need. In this case: continue the rollout (the latency increase traces to an infrastructure sizing artifact of the canary itself, not a code regression), but flag the canary-sizing mismatch as something to fix before the NEXT canary run.

Trade-offs and pitfalls
The temptation under a flat error rate is to assume "no errors means it's fine," but latency regressions are real user-experience problems even without a single error logged, so treating error rate as the only signal that matters is a common and costly mistake. Equally, panicking and rolling back on the FIRST ambiguous signal without doing the diagnostic work wastes the whole point of running a canary, which is to gather enough information to make a confident call rather than a reflexive one.

Direct answer
Deploying from immutable, digest-pinned artifacts means the exact bytes running in production are unambiguous and permanently retrievable: a mutable tag like latest can point to a DIFFERENT image tomorrow than it did today, which means "roll back to the previous version" isn't even a well-defined operation if that version's tag has since been overwritten by something else.

Structured elaboration

• The core problem with mutable tags: if myapp:latest is repointed every time a new build is pushed, there's no way to reliably ask "what was running an hour ago" after the fact, since the tag itself doesn't preserve history, it's a mutable pointer, not a permanent record.
• Digest-pinning solves this: a content digest (a cryptographic hash of the image's actual contents) is immutable by construction, the same digest always refers to the exact same bytes forever; deploying and recording the DIGEST (not just a mutable tag) means "roll back to what was running an hour ago" has a precise, unambiguous answer.
• Reliable rollback: with digest-pinned deploys, a rollback script can always redeploy the exact previous digest with certainty about what it's restoring, versus a mutable-tag-based rollback that might accidentally redeploy something that's since changed under that same tag name.
• Auditability: a digest in your deployment history is a permanent, verifiable record of exactly what ran and when, useful for both operational debugging (what was actually running when this incident happened) and compliance (proving exactly what code was in production at a given point in time).

Worked example
Two deploys both labeled myapp:v2.1 in a system that allows tag reuse: if the tag gets accidentally repushed with a hotfix without bumping the version string, "roll back to v2.1" is now ambiguous, which BUILD does that actually mean? With digest pinning (myapp@sha256:abc123...), there's no such ambiguity; that digest refers to one specific, permanent set of bytes, and a rollback to it is unambiguous regardless of what any mutable tag currently points to.

Trade-offs and pitfalls
Digest-pinning is a small amount of extra discipline (referencing a long hash instead of a friendly tag name in deploy manifests, and often keeping a HUMAN-READABLE tag alongside the digest purely for legibility) for a real, meaningful safety guarantee; the common mistake is using a friendly, mutable tag as the actual deploy reference because it's easier to read, while losing the precise, unambiguous rollback and audit guarantees digest-pinning provides.

Direct answer
Rollback readiness needs both automated tests that run regularly (catching a rollback path that's silently broken, like a script referencing a deprecated command) and periodic manual rehearsals, or gamedays, where a team actually executes a rollback against a realistic environment, because some failure modes (confused communication, a runbook step nobody actually understood, a missing credential) only surface when real people execute the real steps under simulated pressure.

Structured elaboration

• Automated tests: run the actual rollback mechanism (not just verify it exists) against a staging or test environment on a schedule, confirming the command/script still works as the underlying infrastructure evolves; a rollback script referencing a deprecated kubectl flag or an API that's since changed will only be caught this way, not by code review of the runbook text.
• What automated tests can't catch: whether a HUMAN, under time pressure, can actually follow the runbook correctly; whether the escalation path actually reaches the right people; whether the stakeholder-communication templates are usable in the moment; these need a rehearsal with real people, not just a script running unattended.
• Gameday structure: simulate a realistic failure (inject a bad deploy into a staging or isolated environment) and have the on-call rotation execute the ACTUAL rollback runbook, timed and observed, with a debrief afterward on what was confusing, slow, or missing, treated with the same rigor as a real incident's post-incident review.
• Measuring success: time-to-complete for the rollback (is it actually as fast as assumed), how many runbook steps needed clarification or correction during the exercise, whether the person executing it had all the access/credentials they needed without having to request anything mid-exercise.
• Frequency: often quarterly for a critical service's rollback path, or triggered whenever the underlying deployment mechanism changes meaningfully (a platform migration, a new CI/CD tool), since that's exactly when a stale runbook is most likely to have silently drifted out of date.

Worked example
A quarterly gameday for the payments service's rollback runbook: an engineer NOT normally on that service's on-call rotation (deliberately, to test whether the runbook is usable by someone without deep tribal knowledge) executes a simulated rollback against staging, timed at 12 minutes against a target of under 5, with the delay traced to a runbook step referencing a credential the engineer didn't have pre-provisioned. The fix (pre-provisioning that access for the whole on-call rotation, not just the service's usual owners) gets applied and verified in the NEXT gameday, closing the loop rather than just noting it as a finding.

Trade-offs and pitfalls
Gamedays take real time and coordination to run well, which is why teams often let them lapse under normal workload pressure, exactly the wrong instinct, since an untested rollback runbook is a false sense of security that fails you precisely when you need it most. The common mistake is running a gameday once, finding issues, and never re-running it to confirm the fixes actually worked, treating the exercise as a one-time checkbox rather than a recurring discipline.

Direct answer
A progressive-delivery ramp for a payment service needs the automation to actively gate each stage's advance on real metric checks, not just wait out a timer, and the partial-rollback plan for the 50% stage needs to distinguish cleanly between requests that already went through the new code (which may have real side effects, like a payment already processed) and requests still ahead of the rollback taking effect.

Structured elaboration

• 1% canary: the smallest, most cautious stage, watched closely with a shorter observation window since the blast radius is tiny; metric checks focus on error rate and latency deltas against the stable baseline, plus a payment-specific correctness signal (successful-transaction rate, any reconciliation mismatch) since a payment service's most dangerous bugs may not show up as a raw HTTP error at all.
• Ramp to 50% over two hours if clean: this isn't a single jump, it's itself a staged ramp (say 1% -> 10% -> 25% -> 50%, each requiring its own clean metric window before advancing), automated so a human doesn't have to manually approve every micro-step, but with metric checks gating EVERY step, not just the final 50% checkpoint.
• 100% after 24 hours: a long hold at 50% specifically to accumulate enough transaction volume and TIME (payment issues can be slow-building, like a subtle reconciliation drift that only shows up after a batch settlement process runs) before committing to full exposure.
• Partial rollback if problems appear at 50%: reduce the new version's traffic share back down (not necessarily to zero immediately, potentially stepping back to a smaller, still-nonzero percentage to keep gathering diagnostic data on a contained population while you investigate), while the ALREADY-PROCESSED transactions on the new code path need their own review: were any payments processed incorrectly, and do they need a compensating action (a reversal, a manual reconciliation) distinct from the traffic-routing rollback itself?

Worked example
At the 50% stage, an automated check flags a reconciliation discrepancy in a batch of transactions processed by the new code. The traffic-routing rollback (scaling the new version's share back to 5%, not necessarily zero, to preserve some live diagnostic signal) happens within minutes via the automated pipeline. Separately and on a different timeline, a manual reconciliation process reviews every transaction that went through the new code path during its exposure window to determine whether any need a compensating correction, since simply routing future traffic away doesn't undo whatever the already-processed transactions did.

Trade-offs and pitfalls
Payment services are the canonical example of where "roll back the traffic" and "the problem is fixed" are NOT the same thing, since money may have already moved; the automation needs to be scoped clearly to what it CAN fix (stop MORE transactions from hitting the bad path) while explicitly flagging what it can't (undo transactions that already happened), which needs a human-driven reconciliation process rather than being folded into the automated rollback itself.

Direct answer
A good rollback runbook is written so someone under pressure, possibly not the person who wrote it, can execute it correctly without having to reconstruct context: prerequisites to check first, the exact commands (not descriptions of commands), how to verify each step worked, who to notify and when, and a clear escalation path if something doesn't go as expected.

Structured elaboration

• Prerequisites: what needs to be true before you start (do you have the necessary access/credentials, is there a database migration involved that needs its own compatibility check first, is there a specific person who needs to approve an emergency rollback for this particular service).
• Exact commands: copy-pasteable, not paraphrased; "roll back the deployment" is not a runbook step, kubectl rollout undo deployment/checkout-api is. Include the automation link if the rollback is triggered via a CI/CD job rather than a raw command.
• Verification steps: what specifically confirms the rollback worked, both technically (rollout status, pod health) and from a business standpoint (the metric that was degraded has actually recovered), since a rollback that "completes" without the underlying problem resolving means you're not actually done.
• Stakeholder notification: templates, not just "notify stakeholders", a pre-written message for the incident channel, and for customer-facing communication if applicable, so nobody's drafting a message from scratch while also trying to execute a rollback.
• Escalation paths: who to page if the rollback itself fails or if the situation is outside what the runbook covers, with actual names/roles/paging mechanisms, not "escalate as appropriate."
• An emergency-specific section: for the worst case (rollback isn't straightforward, e.g. a migration already ran), a distinct, clearly-labeled set of steps rather than burying emergency guidance inside the normal-case runbook where it's easy to miss under pressure.

Worked example

ROLLBACK RUNBOOK: checkout-api
Prerequisites: confirm no in-flight database migration (check #deploys channel for
  migration status); confirm you have kubectl access to the prod cluster.
Steps:
  1. kubectl rollout undo deployment/checkout-api
  2. kubectl rollout status deployment/checkout-api --timeout=120s
  3. Verify: error rate back under 0.5% on the checkout-api dashboard [link]
  4. Verify: /healthz returns 200 -- curl -sf https://checkout-api.internal/healthz
Notify: post in #incidents using template [link]; page @checkout-oncall if not already engaged.
Escalate to: @senior-sre-oncall if rollout status doesn't complete within 5 minutes,
  or if error rate hasn't recovered within 3 minutes of rollback completing.
Emergency (migration already applied): see EMERGENCY-ROLLBACK.md, do not attempt
  a plain code rollback if a schema migration for this release has already run.

Trade-offs and pitfalls
A runbook that's too generic ("roll back the service, verify it's healthy") provides false confidence, since it doesn't actually reduce the cognitive load on someone executing it under pressure, which is the entire point of having one; a runbook that's never been tested (a "dry run" or gameday rehearsal) risks being subtly wrong or outdated exactly when it's needed most. The most valuable, and most often skipped, sections are the exact verification criteria and the escalation path, both of which people tend to assume are obvious and therefore don't bother writing down.