Safe Deployment and Rollback Strategies Questions

Direct answer
A good rollback runbook is written so someone under pressure, possibly not the person who wrote it, can execute it correctly without having to reconstruct context: prerequisites to check first, the exact commands (not descriptions of commands), how to verify each step worked, who to notify and when, and a clear escalation path if something doesn't go as expected.

Structured elaboration

• Prerequisites: what needs to be true before you start (do you have the necessary access/credentials, is there a database migration involved that needs its own compatibility check first, is there a specific person who needs to approve an emergency rollback for this particular service).
• Exact commands: copy-pasteable, not paraphrased; "roll back the deployment" is not a runbook step, kubectl rollout undo deployment/checkout-api is. Include the automation link if the rollback is triggered via a CI/CD job rather than a raw command.
• Verification steps: what specifically confirms the rollback worked, both technically (rollout status, pod health) and from a business standpoint (the metric that was degraded has actually recovered), since a rollback that "completes" without the underlying problem resolving means you're not actually done.
• Stakeholder notification: templates, not just "notify stakeholders", a pre-written message for the incident channel, and for customer-facing communication if applicable, so nobody's drafting a message from scratch while also trying to execute a rollback.
• Escalation paths: who to page if the rollback itself fails or if the situation is outside what the runbook covers, with actual names/roles/paging mechanisms, not "escalate as appropriate."
• An emergency-specific section: for the worst case (rollback isn't straightforward, e.g. a migration already ran), a distinct, clearly-labeled set of steps rather than burying emergency guidance inside the normal-case runbook where it's easy to miss under pressure.

Worked example

ROLLBACK RUNBOOK: checkout-api
Prerequisites: confirm no in-flight database migration (check #deploys channel for
  migration status); confirm you have kubectl access to the prod cluster.
Steps:
  1. kubectl rollout undo deployment/checkout-api
  2. kubectl rollout status deployment/checkout-api --timeout=120s
  3. Verify: error rate back under 0.5% on the checkout-api dashboard [link]
  4. Verify: /healthz returns 200 -- curl -sf https://checkout-api.internal/healthz
Notify: post in #incidents using template [link]; page @checkout-oncall if not already engaged.
Escalate to: @senior-sre-oncall if rollout status doesn't complete within 5 minutes,
  or if error rate hasn't recovered within 3 minutes of rollback completing.
Emergency (migration already applied): see EMERGENCY-ROLLBACK.md, do not attempt
  a plain code rollback if a schema migration for this release has already run.

Trade-offs and pitfalls
A runbook that's too generic ("roll back the service, verify it's healthy") provides false confidence, since it doesn't actually reduce the cognitive load on someone executing it under pressure, which is the entire point of having one; a runbook that's never been tested (a "dry run" or gameday rehearsal) risks being subtly wrong or outdated exactly when it's needed most. The most valuable, and most often skipped, sections are the exact verification criteria and the escalation path, both of which people tend to assume are obvious and therefore don't bother writing down.

Direct answer
A post-deploy health-poll script watches the new version for a fixed window right after it starts receiving traffic, aggregating success rate and latency, and explicitly marks the deployment failed (triggering whatever the pipeline's next step is, typically an automatic rollback) if either crosses a threshold, rather than assuming silence means success.

Structured elaboration and worked example (logic verified)

import time
import requests

def poll_health(url: str, duration_seconds: int = 300, interval_seconds: int = 5,
                 success_rate_threshold: float = 0.99, latency_threshold_ms: float = 500):
    # Polls a health endpoint for `duration_seconds`, aggregating results.
    # Returns (passed: bool, summary: dict).
    start = time.monotonic()
    successes = 0
    total = 0
    latencies = []

    while time.monotonic() - start < duration_seconds:
        req_start = time.monotonic()
        try:
            resp = requests.get(url, timeout=5)
            elapsed_ms = (time.monotonic() - req_start) * 1000
            latencies.append(elapsed_ms)
            total += 1
            if resp.status_code == 200:
                successes += 1
        except requests.RequestException:
            total += 1
            latencies.append(latency_threshold_ms * 10)  # count a hard failure as very slow, not silently dropped
        time.sleep(interval_seconds)

    if total == 0:
        return False, {"reason": "no requests completed, cannot assess health"}

    success_rate = successes / total
    avg_latency = sum(latencies) / len(latencies)
    passed = success_rate >= success_rate_threshold and avg_latency <= latency_threshold_ms
    return passed, {"success_rate": success_rate, "avg_latency_ms": avg_latency, "requests": total}

Key structural decisions: a network exception is counted as a failed, slow request rather than silently skipped, since a connection timeout is itself a strong negative signal that shouldn't be excluded from the aggregate just because it didn't return an HTTP status code at all. Zero completed requests is treated as an explicit failure with a clear reason, never silently passed, since "we don't have enough data" should never be mistaken for "it's healthy."

Trade-offs and pitfalls
A fixed polling interval (5 seconds here) trades responsiveness for load on the health endpoint; a service under real stress from a bad deploy doesn't need to be hammered by an aggressive health-check loop on top of everything else. The retry/timeout handling matters more than it looks: a script that raises an unhandled exception on the FIRST network hiccup, rather than counting it as a data point and continuing, produces a false "script crashed" result instead of the actually useful "service is unhealthy" signal the pipeline needs to act on.

Direct answer
Linear, exponential, and custom ramp schedules trade off rollout speed against how much time you spend at each risk level, and the right choice depends on how confident you already are in the change and how quickly a problem would need to surface to be caught.

Structured elaboration

• Linear (e.g., 5% -> 10% -> 15% ... -> 100% in equal steps): predictable, easy to reason about and communicate, but spends the same amount of time at every stage regardless of how much confidence you've already accumulated, which can be needlessly slow for later, higher-confidence stages.
• Exponential (e.g., 1% -> 5% -> 25% -> 100%): spends the MOST time and caution at the smallest, riskiest early stages where a bug affects the fewest users, and moves faster once confidence has built, which usually matches the actual risk profile better than linear: the marginal value of extra caution shrinks as you've already validated more traffic.
• Custom schedules: useful when you have a specific constraint a standard curve doesn't fit, for example a traffic-pattern reason to avoid crossing 50% right before a known peak-traffic period, or a business reason to hold at a specific percentage for a fixed calendar window (a scheduled review meeting, a compliance sign-off gate).
• Encoding it in an automated tool: most progressive-delivery controllers (Argo Rollouts, Flagger) accept a declarative list of steps, each with a percentage and either a fixed pause duration or a condition (an automated analysis run) that must pass before advancing to the next step, so the schedule itself is just configuration, not custom code, in the common case.

Worked example

steps:
  - setWeight: 5
    pause: {duration: 10m}
  - setWeight: 25
    pause: {duration: 15m}
  - setWeight: 50
    pause: {duration: 15m}
  - setWeight: 100

This is roughly exponential in spirit: cautious early (10 minutes at just 5%), then faster once early confidence is built (15 minutes at 25% and 50% before going fully live). A team with a very well-tested, low-risk change might compress this to just two steps (10% for 5 minutes, then 100%), while a team shipping a high-risk payment-logic change might insert several MORE steps and longer pauses, or add an explicit manual-approval gate between 25% and 50%.

Trade-offs and pitfalls
A schedule that's too aggressive early defeats the purpose of canarying at all, since a bug at 25% affects a genuinely large user population before you'd catch it; one that's too conservative throughout slows every release regardless of how well-tested or low-risk the change actually is, training the team to see the canary process as pure friction. The strongest programs vary the schedule by the assessed risk of the specific change rather than using one fixed schedule for everything.

Direct answer
Deploying from immutable, digest-pinned artifacts means the exact bytes running in production are unambiguous and permanently retrievable: a mutable tag like latest can point to a DIFFERENT image tomorrow than it did today, which means "roll back to the previous version" isn't even a well-defined operation if that version's tag has since been overwritten by something else.

Structured elaboration

• The core problem with mutable tags: if myapp:latest is repointed every time a new build is pushed, there's no way to reliably ask "what was running an hour ago" after the fact, since the tag itself doesn't preserve history, it's a mutable pointer, not a permanent record.
• Digest-pinning solves this: a content digest (a cryptographic hash of the image's actual contents) is immutable by construction, the same digest always refers to the exact same bytes forever; deploying and recording the DIGEST (not just a mutable tag) means "roll back to what was running an hour ago" has a precise, unambiguous answer.
• Reliable rollback: with digest-pinned deploys, a rollback script can always redeploy the exact previous digest with certainty about what it's restoring, versus a mutable-tag-based rollback that might accidentally redeploy something that's since changed under that same tag name.
• Auditability: a digest in your deployment history is a permanent, verifiable record of exactly what ran and when, useful for both operational debugging (what was actually running when this incident happened) and compliance (proving exactly what code was in production at a given point in time).

Worked example
Two deploys both labeled myapp:v2.1 in a system that allows tag reuse: if the tag gets accidentally repushed with a hotfix without bumping the version string, "roll back to v2.1" is now ambiguous, which BUILD does that actually mean? With digest pinning (myapp@sha256:abc123...), there's no such ambiguity; that digest refers to one specific, permanent set of bytes, and a rollback to it is unambiguous regardless of what any mutable tag currently points to.

Trade-offs and pitfalls
Digest-pinning is a small amount of extra discipline (referencing a long hash instead of a friendly tag name in deploy manifests, and often keeping a HUMAN-READABLE tag alongside the digest purely for legibility) for a real, meaningful safety guarantee; the common mistake is using a friendly, mutable tag as the actual deploy reference because it's easier to read, while losing the precise, unambiguous rollback and audit guarantees digest-pinning provides.

Direct answer
When B must roll back but A stays on its new (backward-compatible) version, the core problem is that B's incompatible writes may have already happened before you detect the issue, so the fix isn't just "redeploy old B," it's identifying and correcting whatever inconsistent state those writes left behind, while B's rollback itself is straightforward since A's compatibility means A doesn't need to change at all.

Structured elaboration

1. Stop further damage: roll back B's code immediately (this part is simple exactly because A is backward-compatible with B's old version, no coordination needed there), which stops NEW incompatible writes from happening, but doesn't undo ones that already occurred.
2. Identify what's actually inconsistent: determine which writes B made in its brief new-version window were the problematic, incompatible ones, versus which writes (even during that window) were fine; this typically requires either an audit log of what changed, or a way to distinguish "written by new B" from "written by old B" (a version marker on the data itself, if you have one, or inferring from timestamps against the deploy window).
3. In-flight requests specifically: any request that started against new-B logic but hasn't yet completed when the rollback happens needs explicit handling, either let it finish against the code it started with (avoiding a mid-request logic switch) and then reconcile its result afterward if needed, or, if that's not safe, actively cancel/retry it against the now-rolled-back old B.
4. Reconciliation toward eventual consistency: for the identified inconsistent writes, either a compensating action (a corrective write that brings the data back to what old-B's logic would have produced) or, if the incompatible writes are numerous and hard to individually correct, a broader reconciliation job that recomputes affected records from source data, run once B is confirmed stable on the old version again.
5. Idempotency throughout: both the rollback itself and any compensating/reconciliation actions need to be safely re-runnable, since this kind of recovery work is exactly the scenario where a retry (from a nervous on-call engineer, or from an automated retry mechanism) is likely.

Worked example
Service B's new version wrote records in a new, incompatible format for roughly 8 minutes before the regression was caught and B rolled back. An audit log (or a version-tagged field on the written records) identifies exactly which records fall in that window; a reconciliation job re-derives the correct, old-format value for each of those specific records from upstream source data, run idempotently (safe to re-trigger if it's interrupted partway through) so a retry doesn't double-apply the correction.

Trade-offs and pitfalls
The scope of this problem is directly proportional to how LONG the incompatible version was live before detection, which is a strong argument for the fast, automated rollback-trigger discipline covered elsewhere in this topic; a regression caught in 30 seconds via automated detection leaves a much smaller reconciliation problem than one caught 30 minutes later via a human noticing something looked off. The common mistake is treating the code rollback as the end of the incident, when the DATA reconciliation is often the harder, longer-running part of actually resolving it.