Safe Deployment and Rollback Strategies Questions
Releasing changes to production safely and incrementally, and recovering when they fail: blue-green, canary, and rolling deployments, feature flags, dark launches, traffic shifting, and progressive rollout, together with rollback strategies, safe-deploy practices, blast-radius containment, automated recovery, and safe forward/backward migration. Covers deployment orchestration across cloud platforms, staged exposure of new behavior to users, assessing deployment risk, designing reversible releases, and restoring a known-good state quickly. Focuses on how a release reaches production and how it is unwound on failure, distinct from broader incident command, which lives in Enterprise Operations & Incident Management.
What is a smoke test in the context of a deployment, and how is it different from a full integration-test suite? What would a minimal automated smoke-test gate look like right before a deploy is promoted?
Sample Answer
Direct answer
A smoke test is a small, fast set of checks run immediately after a deploy to catch catastrophic breakage (the service won't start, a core endpoint 500s, a critical dependency is unreachable) before you either promote a canary or declare the deploy successful. It's deliberately much narrower than a full integration-test suite, which exercises business logic in depth but takes far longer to run, too long to gate a live production promotion decision on.
Structured elaboration
- What it checks: is the service up and responding, do the 2-3 most critical endpoints return a sane response, can it reach its essential dependencies (database, cache, key downstream services)? Not edge cases, not exhaustive business-logic coverage.
- Where it runs: right after the new version starts receiving traffic (or right after it's deployed but before traffic ramps, if you can health-check it in isolation first), as a hard gate before the rollout is allowed to proceed further.
- Speed matters more than coverage here: a smoke test that takes 10 minutes defeats the point of a fast-fail gate; it should complete in seconds to low single-digit minutes.
- A minimal suite design: (1) hit a
/healthzendpoint and confirm a 200, (2) exercise one real, representative request through the core business path (not a synthetic no-op), and confirm the response shape is correct, (3) confirm connectivity to the database and any hard-dependency downstream service, often by checking that the health endpoint itself reports THEIR health, not by hitting them directly.
Worked example
For an e-commerce checkout service: smoke tests would be (1) GET /healthz returns 200, (2) POST a test order through checkout in a sandboxed test-account mode and confirm a valid order-confirmation response, (3) confirm the health endpoint reports the payment-processor dependency as reachable. If any of these fail, the pipeline halts the rollout automatically before more traffic reaches the new version, well before a full integration suite (which might separately test discount codes, tax calculation, inventory edge cases) would even finish running.
Trade-offs and pitfalls
Too narrow a smoke test misses real breakage (a service that responds 200 to health checks but is silently broken for its actual purpose); too broad and it becomes slow enough to defeat the purpose of a fast gate, effectively becoming a second integration suite. The common mistake is a smoke test that only checks liveness (is the process up) without exercising any real business logic, which catches crashes but misses the more common and more dangerous case of a service that's up and responding but functionally wrong.
Design a progressive-delivery ramp for a payment service: an initial 1% canary, ramp to 50% over two hours if clean, then 100% after 24 hours. What automation and metric checks run at each stage, and how do you handle a partial rollback if problems appear at the 50% stage?
Sample Answer
Direct answer
A progressive-delivery ramp for a payment service needs the automation to actively gate each stage's advance on real metric checks, not just wait out a timer, and the partial-rollback plan for the 50% stage needs to distinguish cleanly between requests that already went through the new code (which may have real side effects, like a payment already processed) and requests still ahead of the rollback taking effect.
Structured elaboration
- 1% canary: the smallest, most cautious stage, watched closely with a shorter observation window since the blast radius is tiny; metric checks focus on error rate and latency deltas against the stable baseline, plus a payment-specific correctness signal (successful-transaction rate, any reconciliation mismatch) since a payment service's most dangerous bugs may not show up as a raw HTTP error at all.
- Ramp to 50% over two hours if clean: this isn't a single jump, it's itself a staged ramp (say 1% -> 10% -> 25% -> 50%, each requiring its own clean metric window before advancing), automated so a human doesn't have to manually approve every micro-step, but with metric checks gating EVERY step, not just the final 50% checkpoint.
- 100% after 24 hours: a long hold at 50% specifically to accumulate enough transaction volume and TIME (payment issues can be slow-building, like a subtle reconciliation drift that only shows up after a batch settlement process runs) before committing to full exposure.
- Partial rollback if problems appear at 50%: reduce the new version's traffic share back down (not necessarily to zero immediately, potentially stepping back to a smaller, still-nonzero percentage to keep gathering diagnostic data on a contained population while you investigate), while the ALREADY-PROCESSED transactions on the new code path need their own review: were any payments processed incorrectly, and do they need a compensating action (a reversal, a manual reconciliation) distinct from the traffic-routing rollback itself?
Worked example
At the 50% stage, an automated check flags a reconciliation discrepancy in a batch of transactions processed by the new code. The traffic-routing rollback (scaling the new version's share back to 5%, not necessarily zero, to preserve some live diagnostic signal) happens within minutes via the automated pipeline. Separately and on a different timeline, a manual reconciliation process reviews every transaction that went through the new code path during its exposure window to determine whether any need a compensating correction, since simply routing future traffic away doesn't undo whatever the already-processed transactions did.
Trade-offs and pitfalls
Payment services are the canonical example of where "roll back the traffic" and "the problem is fixed" are NOT the same thing, since money may have already moved; the automation needs to be scoped clearly to what it CAN fix (stop MORE transactions from hitting the bad path) while explicitly flagging what it can't (undo transactions that already happened), which needs a human-driven reconciliation process rather than being folded into the automated rollback itself.
What's the difference between a rollback (redeploying the previous artifact) and a revert (a new forward commit that undoes the change)? Which would you reach for after discovering a production regression, and why?
Sample Answer
Direct answer
A rollback redeploys the previous, already-tested version of the artifact; a revert is a NEW forward commit that undoes the change in source control and then gets built and deployed like any other change. After discovering a production regression, rollback is almost always the faster, safer first move, since it restores a known-good state immediately, while a revert (even though it also "undoes" the change conceptually) still has to go through the normal build-and-deploy pipeline before it takes effect.
Structured elaboration
- Rollback: uses infrastructure/deployment tooling (redeploy the previous artifact,
kubectl rollout undo, switch a blue-green environment back) to restore the PREVIOUS RUNNING STATE directly, without rebuilding anything; it's fast precisely because the previous version is already built, tested, and known-good. - Revert: a source-control operation (
git revert) that creates a new commit undoing the change; this new commit then needs to go through CI, build, and deploy like any normal change, which takes real time even if every step passes cleanly, and it's not automatically faster just because it "undoes" something. - When you'd reach for each: rollback for the immediate, fast restoration of service; revert as the FOLLOW-UP action that keeps the source-control history clean and honest about what's actually running, and as the mechanism for making the "undo" permanent once you've confirmed the rollback fixed the problem (otherwise the next normal deploy, built from a source tree that still contains the bad change, would silently reintroduce the regression).
- Why both matter, not just one: rolling back WITHOUT eventually reverting means the next deploy from the current source tree reintroduces the bug, since the source code still contains the bad change even though the RUNNING version has been reverted; reverting without rolling back first means waiting through a full build-and-deploy cycle before service actually recovers, when a faster path was available.
Worked example
A regression discovered five minutes after a deploy: immediately kubectl rollout undo restores the previous, known-good version running in production within seconds. Separately, and not blocking that fast recovery, git revert <bad-commit> is pushed to keep the source tree consistent with what's actually running, so the next unrelated deploy (which will build from the current source tree) doesn't accidentally reintroduce the regression.
Trade-offs and pitfalls
The common mistake is treating these as interchangeable or doing only one: rolling back without ever reverting leaves a latent landmine in the source tree that resurfaces on the next deploy; reverting without rolling back first needlessly extends the outage while waiting for a full pipeline run when a faster path existed. The strongest practice is rollback FIRST for immediate recovery, revert SECOND (often within the same incident) to make the fix permanent in source control.
Define deployment frequency, mean time to recovery, and change-failure-rate, the DORA-style metrics used to gauge deployment health and velocity. How would a team measure each, and what's a reasonable target for a high-performing team?
Sample Answer
Direct answer
Deployment frequency (how often you ship to production), mean time to recovery (how fast you restore service after an incident), and change-failure-rate (what fraction of deployments cause a production problem) are three of the four DORA metrics that together describe how fast AND how safely a team ships. High-performing teams deploy often, recover fast, and fail rarely; the point of tracking all three together is that any one alone can be gamed or misleading.
Structured elaboration
- Deployment frequency: count of production deploys per day/week per service. Elite teams deploy on-demand, often multiple times a day; measuring it is usually just counting CI/CD pipeline "deploy to prod" events.
- Mean time to recovery (MTTR): from the moment an incident starts degrading users to the moment service is restored. Measuring it accurately requires two reliable timestamps: incident START (usually from the first alert or the first bad metric) and RESOLVED (usually from the alert clearing or an explicit "resolved" marker), which is harder to instrument well than it sounds, since teams often only record when the fix was DEPLOYED, not when the SERVICE actually recovered.
- Change-failure-rate: percentage of deployments that require a rollback, hotfix, or cause an incident, out of total deployments. Requires tagging deployments with an outcome, which usually means linking your deploy log to your incident/rollback log.
- Reasonable targets (per DORA's own research bands): elite performers deploy on-demand (multiple times per day), recover in under an hour, and keep change-failure-rate under 15%. Teams earlier in their DevOps maturity might deploy weekly to monthly, take a day or more to recover, and see failure rates well above that.
Worked example
A team ships 12 times a week, has 2 of those deploys cause an incident requiring rollback (change-failure-rate ~17%), and those 2 incidents took 25 and 40 minutes respectively to resolve (MTTR ~33 minutes). That profile is roughly "high" performing on frequency and recovery speed but borderline on failure rate, suggesting the team's canary/testing discipline needs tightening before pushing frequency even higher.
Trade-offs and pitfalls
Optimizing deployment frequency alone, without watching change-failure-rate, just means shipping more bugs faster; the metrics are meant to be read together. The most common measurement pitfall is MTTR calculated from "code fix deployed" instead of "users stopped being affected," which systematically understates real recovery time whenever a rollback or mitigation restores service before the actual fix ships.
Given a dependency graph of interdependent microservices that may include cycles, design an algorithm to compute a safe rollback order: safe parallel batches, handling cycles, and respecting compatibility constraints.
Sample Answer
Direct answer
Computing a safe rollback order across a dependency graph that may include cycles means treating it as a graph problem: find groups of services that can safely roll back together in parallel (respecting who depends on whom), and specifically handle cycles by either breaking them at a deliberately-chosen weak point or treating a cyclic group as a single atomic rollback unit, since a true cycle has no valid strict ordering on its own.
Structured elaboration
- Model the dependency graph: an edge from service X to Y means X depends on Y's CURRENT (new) behavior; rolling back Y before X could break X, so the safe rollback order is generally the REVERSE of dependency direction, dependents roll back before their dependencies, mirroring the general partial-rollback-ordering principle used elsewhere in this topic.
- Topological sort for the acyclic portion: for a dependency graph with no cycles, a standard topological sort (repeatedly pick nodes with no remaining incoming "depends on me" edges) gives a valid ordering, and services with no dependency relationship to each other at a given point in the sort can roll back in PARALLEL, safely, as a batch.
- Handling cycles: a genuine cycle (X depends on Y's new behavior, Y depends on X's) has no valid strict ordering, since rolling back either one first breaks the other; the practical resolution is either (a) treat the entire cyclic group as ONE atomic rollback unit, rolling all of them back together simultaneously so neither is ever left depending on the other's now-reverted-but-not-yet-reverted state, or (b) if one edge in the cycle is weaker/less critical than the other (say, one direction only affects a non-critical code path), deliberately break the cycle there and accept a brief, bounded inconsistency on that specific weaker dependency during the transition.
- Compatibility constraints beyond pure ordering: even with a valid order, each individual rollback step still needs the underlying compatibility check (is old code X compatible with whatever state Y, still on its new version, is currently in) covered elsewhere in this topic; graph ordering alone doesn't guarantee compatibility, it just determines a safe SEQUENCE to check and execute rollbacks in.
Worked example
flowchart LR
A --> B
B --> C
C --> D
D --> B
Services A, B, C, D, where B, C, D form a cycle (B depends on D depending on C depending on B) and A depends on B. The safe order: A rolls back first (nothing depends on A). Then the B-C-D cycle, having no valid strict internal ordering, rolls back as a single atomic batch, all three simultaneously, rather than attempting to sequence them individually.
Trade-offs and pitfalls
Treating a cyclic group as one atomic unit is operationally more complex (you need all three rollback mechanisms to succeed together, or handle a partial-failure-within-the-atomic-group case, which is itself a hard problem) but it's the only approach that doesn't introduce a real compatibility gap somewhere in the cycle; the common mistake is picking an arbitrary order within a cycle without recognizing it's a cycle at all, which can silently leave one service depending on another's already-reverted-but-not-yet-caught-up state during the transition.
Unlock Full Question Bank
Get access to all Safe Deployment and Rollback Strategies interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.