Safe Deployment and Rollback Strategies Questions

Direct answer
A full rollback reverts every service or component involved in a release back to its previous version; a partial rollback reverts only some of them, leaving others on the new version. You choose partial when only some of the changed components are actually implicated in the problem and reverting the rest would be unnecessary churn or would itself introduce risk (for example, if a later component now depends on an earlier one's new behavior).

Structured elaboration

• When full rollback makes sense: a single service deployed a bad change, or several services were released together as one coupled unit where partial reversion would leave them in an inconsistent, untested combination.
• When partial makes sense: a coordinated multi-service release where only ONE service's new version is misbehaving, and the others are backward-compatible enough to keep running against either the old or new version of the problem service.
• Risk in partial rollback: you have to be confident the services you're leaving on the new version don't assume the now-reverted service's new behavior; if they do, a partial rollback trades one outage for a different, subtler one.

Worked example
Full: three tightly coupled microservices (auth, session, and API-gateway) deploy together as one release because the API gateway's new version assumes session's new token format; a bug anywhere means reverting all three together, since a partial revert would leave a token-format mismatch between them.
Partial: a release touches a recommendations service and a completely independent notifications service in the same deploy window purely for scheduling convenience; if only recommendations regresses, rolling back just that service and leaving notifications on its (unrelated, unaffected) new version is the right call, since reverting notifications too would be pure unnecessary churn.

Trade-offs and pitfalls
Partial rollback is faster and less disruptive when it's safe, but the judgment call of "is it actually safe to leave X on the new version" is exactly where mistakes happen; teams that don't explicitly map service dependencies before a coordinated release often discover the hard way, mid-incident, that two "independent" services weren't as independent as assumed.

Direct answer
A strong answer here follows STAR: what tipped you off that something was wrong, what you actually did to execute the rollback, and what changed afterward so the same failure mode doesn't recur. The interviewer is listening for concrete detection signals and concrete actions, not a vague "we noticed issues and rolled back."

Structured elaboration

• Situation/Task: name the service, the scale (traffic volume matters for how fast things degraded), and what the deploy changed.
• Action - detection: was it a dashboard alert, a customer report, a synthetic check? Specificity here (a named metric crossing a named threshold) is what separates a real story from a generic one.
• Action - execution: what commands or automation did you actually run? Redeploy previous image tag, flip a feature flag, revert a config? Did you have to coordinate a database rollback too, or was code-only sufficient?
• Action - safety checks: how did you confirm the rollback itself was safe before running it (was there a schema dependency you had to check first)?
• Result: how long did it take from detection to resolution, and what was the actual customer impact?
• Follow-up: what changed afterward: a new automated rollback trigger, a canary gate that would have caught it earlier, a runbook that didn't exist before?

Worked example
"We shipped a change to our checkout service that introduced a null-pointer path under a rare cart configuration. Fifteen minutes after full rollout, our error-rate alert fired at 3% (baseline 0.1%). I confirmed via the dashboard the spike started at the deploy timestamp, then ran our rollback script to redeploy the previous image tag, which took about ninety seconds including health-check verification. Error rate returned to baseline within two minutes of the redeploy completing. Afterward we added that cart configuration as an explicit test case and lowered our canary's automated error-rate threshold so a similar regression would be caught at 1% traffic instead of 100%."

Trade-offs and pitfalls
A common weak answer stops at "we rolled back and it was fixed" without naming a detection signal or a concrete command, which reads as secondhand rather than lived experience. Another common gap is skipping the "what changed afterward" beat entirely, which is often what the interviewer is most interested in, since it signals whether you learn from incidents systemically or just fight fires one at a time.

Direct answer
Protecting an executive-facing dashboard from a canary-stage data-pipeline or metric-definition change means keeping the canary's output entirely separate from what feeds the actual reporting surface until it's validated, rather than letting a partially-rolled-out change silently blend into aggregate numbers that decision-makers are actively looking at.

Structured elaboration

• Shadow the change before it touches real reports: run the new pipeline logic or metric definition in PARALLEL, writing its output to a separate, clearly-labeled location (a staging table, a "candidate" version of the metric) rather than directly into the table or dashboard executives already see, so nothing user-facing changes until validation is deliberately complete.
• Feature-flag the METRIC DEFINITION itself, not just the code: if the change is a redefinition of how a metric is computed, gate which definition is ACTIVE for reporting purposes behind an explicit flag, so a canary-stage computation issue can't accidentally leak a wrong number into a report just because the underlying job happened to run.
• Validate against known-good historical values first: before any canary output is trusted even in a staging location, recompute a PAST period's numbers with the new logic and confirm they match the already-published, already-trusted historical values (within an expected, explainable tolerance if the change is an intentional definitional improvement); a canary computing a number for the future is much harder to sanity-check than one that can be validated against a period whose "correct" answer is already known.
• Explicit sign-off before promoting the new definition to the live dashboard: someone (a data/analytics owner, not just the engineer who wrote the pipeline change) reviews the validated candidate output against the historical baseline and the parallel-run comparison before the flag flips the live dashboard over to the new definition, since a wrong number reaching an executive dashboard has outsized organizational cost (bad decisions made on bad data) compared to a typical user-facing bug.
• Rollback if a leak is detected anyway: since dashboards are often looked at asynchronously (not real-time monitored the way a service's error rate is), detecting a leaked bad number might happen HOURS after it occurred; the rollback plan needs an explicit correction/republish step, not just "revert the code," since the WRONG number may have already been seen, screenshotted, or acted on by a stakeholder before anyone caught it.

Worked example
A change to how "monthly active users" is computed runs in shadow for the past three completed months, and its output is compared against the already-published MAU figures for those months; discrepancies beyond an expected, explainable tolerance (say, more than 0.5%, given the change is a bug fix expected to shift the number slightly, not dramatically) block promotion and trigger investigation. Once validated, the metric-definition flag flips for the CURRENT month's live dashboard, with the old definition kept computable in parallel for a further period specifically so any late-discovered discrepancy can be diagnosed against a known-good comparison.

Trade-offs and pitfalls
The strongest protection here (shadow computation plus historical-value validation plus explicit human sign-off) is meaningfully slower than a normal application canary's automated promote/rollback cycle, which is the right trade for something executives make real decisions from, but would be excessive overhead for a low-stakes internal metric nobody's making consequential decisions based on; matching the rigor to the actual stakes of the specific metric or dashboard is the real judgment call. The common mistake is treating a metrics-pipeline canary exactly like an application-code canary (a quick traffic-split-and-watch-error-rate check), missing that the FAILURE MODE here (a plausible-looking but wrong number silently reaching a decision-maker) doesn't show up in the technical metrics a normal canary watches at all.

Direct answer
Blue-green, canary, and rolling all reduce the risk of a bad release, but through different mechanisms: blue-green switches ALL traffic at once between two full environments, canary exposes a small SLICE of traffic to the new version before widening it, and rolling replaces instances gradually IN PLACE. A plain recreate deployment, by contrast, tears down the old version entirely before starting the new one, accepting downtime in exchange for simplicity.

Structured elaboration

Rollback complexity nuance: blue-green's rollback is fastest because the old environment never stopped running; canary and rolling both have to actively redeploy or re-route, which takes real time even if it's automated; recreate's "rollback" is simple mechanically but means accepting a second period of downtime.

Stateful services: all three of blue-green/canary/rolling get significantly harder with state (a database, in-memory session data, local disk), because you can't just duplicate or partially expose the data layer the way you can stateless compute; the deployment strategy for the STATELESS layer often decouples from a separate, more careful strategy for the DATA layer.

Worked example
A stateless API fronting a shared database: canary is a strong default, since it limits blast radius on the code change while the shared database (which doesn't get canaried the same way) stays constant underneath. Blue-green would be a better fit if the team's top priority is minimizing time-to-rollback over minimizing blast radius, since flipping back to the old environment is close to instant.

Trade-offs and pitfalls
There's no universally "best" strategy: the choice trades off blast radius, rollback speed, infrastructure cost, and operational complexity, and the right answer depends on which of those the specific service and change profile cares about most. A common mistake is picking a strategy based on what's trendy (everyone reaches for canary) rather than what the actual risk profile of the change calls for; a low-risk config change might not need any of this ceremony at all.

Direct answer
A rolling update replaces old-version instances with new-version ones gradually, a few at a time, so the service stays available with a mix of old and new versions running simultaneously during the transition. A recreate deployment, by contrast, terminates ALL old instances first and only then starts the new ones, which means a period of full downtime but avoids ever running mixed versions.

Structured elaboration
For a stateless microservice in Kubernetes, a rolling update:

1. Kubernetes creates a batch of new-version pods (controlled by maxSurge, how many extra pods above the target replica count are allowed).
2. Waits for those new pods to pass their readiness probe before routing traffic to them.
3. Terminates an equivalent batch of old-version pods (controlled by maxUnavailable, how many pods can be down at once).
4. Repeats until all pods are on the new version.

Common failure modes to watch for during a rollout:

• Readiness probe misconfigured too loosely: traffic gets routed to a pod that's technically "ready" but not actually able to serve correctly yet (cache not warmed, connection pool not established).
• Version skew during the mixed-version window: old and new pods running simultaneously both talk to the same downstream dependencies (shared database, shared cache), so if the new version isn't backward-compatible with what the old version expects, you get intermittent failures purely from which version happened to handle a given request.
• Resource exhaustion from surge: if maxSurge allows too many extra pods at once relative to available cluster capacity, new pods can fail to schedule, stalling the rollout partway.
• A bad new version rolling out gradually still affects a growing fraction of traffic before anyone notices, unlike blue-green where the bad version is fully isolated until an explicit cutover.

Worked example
A 20-replica deployment with maxSurge: 25% and maxUnavailable: 25% creates up to 5 extra pods (25 total temporarily) while taking down up to 5 old pods at a time, cycling through until all 20 are on the new version. If the new version has a subtle bug that only manifests under a specific downstream response, roughly a quarter of traffic is exposed to it at any point mid-rollout, growing toward 100% as the rollout proceeds, unless something halts it.

Trade-offs and pitfalls
Rolling update avoids downtime and extra infrastructure cost (no duplicate fleet, unlike blue-green) but accepts a mixed-version window where compatibility between old and new has to hold, and it doesn't isolate a bad release the way canary or blue-green does; it just gradually replaces capacity regardless of whether the new version is actually healthy, UNLESS combined with a readiness-probe-based or metrics-based halt condition.