Safe Deployment and Rollback Strategies Questions

Direct answer
A rollback redeploys the previous, already-tested version of the artifact; a revert is a NEW forward commit that undoes the change in source control and then gets built and deployed like any other change. After discovering a production regression, rollback is almost always the faster, safer first move, since it restores a known-good state immediately, while a revert (even though it also "undoes" the change conceptually) still has to go through the normal build-and-deploy pipeline before it takes effect.

Structured elaboration

• Rollback: uses infrastructure/deployment tooling (redeploy the previous artifact, kubectl rollout undo, switch a blue-green environment back) to restore the PREVIOUS RUNNING STATE directly, without rebuilding anything; it's fast precisely because the previous version is already built, tested, and known-good.
• Revert: a source-control operation (git revert) that creates a new commit undoing the change; this new commit then needs to go through CI, build, and deploy like any normal change, which takes real time even if every step passes cleanly, and it's not automatically faster just because it "undoes" something.
• When you'd reach for each: rollback for the immediate, fast restoration of service; revert as the FOLLOW-UP action that keeps the source-control history clean and honest about what's actually running, and as the mechanism for making the "undo" permanent once you've confirmed the rollback fixed the problem (otherwise the next normal deploy, built from a source tree that still contains the bad change, would silently reintroduce the regression).
• Why both matter, not just one: rolling back WITHOUT eventually reverting means the next deploy from the current source tree reintroduces the bug, since the source code still contains the bad change even though the RUNNING version has been reverted; reverting without rolling back first means waiting through a full build-and-deploy cycle before service actually recovers, when a faster path was available.

Worked example
A regression discovered five minutes after a deploy: immediately kubectl rollout undo restores the previous, known-good version running in production within seconds. Separately, and not blocking that fast recovery, git revert <bad-commit> is pushed to keep the source tree consistent with what's actually running, so the next unrelated deploy (which will build from the current source tree) doesn't accidentally reintroduce the regression.

Trade-offs and pitfalls
The common mistake is treating these as interchangeable or doing only one: rolling back without ever reverting leaves a latent landmine in the source tree that resurfaces on the next deploy; reverting without rolling back first needlessly extends the outage while waiting for a full pipeline run when a faster path existed. The strongest practice is rollback FIRST for immediate recovery, revert SECOND (often within the same incident) to make the fix permanent in source control.

Direct answer
Server-side evaluation is generally safer for a rollout because the server can change a flag's value instantly and consistently for every client, while client-side evaluation depends on each device having up-to-date flag configuration, which is exactly what breaks down under intermittent connectivity.

Structured elaboration

• Client-side evaluation: the app itself decides whether a feature is on, usually based on a flag configuration it fetched and cached at some point. Faster to evaluate at request time (no network round-trip needed) and works fully offline, but a device on a stale cache keeps using an OLD flag value until it next successfully syncs, which for an intermittently-connected mobile app could be minutes to hours out of date.
• Server-side evaluation: the server decides on each request. Always reflects the current flag state instantly and consistently, but requires a live connection for every decision that depends on the flag, which is a problem for a mobile app trying to work offline or under a flaky connection.
• Default behavior when a flag can't be fetched: this is the crux of the safety question. The system needs an explicit, deliberately chosen default for "no data available," and that default should almost always be the SAFE, conservative behavior (feature OFF, or the old known-good code path), never "assume the last cached value is still correct" for anything risk-sensitive, since an unreachable device might be running a stale cache for an unknown, possibly long, period.

Worked example
A payment-related feature flag on a mobile app with client-side evaluation: the SDK is configured so that if it can't reach the flag service within a short timeout, it falls back to a hardcoded, safe default (feature OFF) rather than serving whatever was last cached, which might be hours or days stale if the device has been offline. This costs some feature-availability under poor connectivity (some users see the old behavior more often than strictly necessary) in exchange for never risking a stale, potentially-unsafe flag state driving payment logic.

Trade-offs and pitfalls
A hybrid is common in practice: client-side evaluation for speed and offline resilience, paired with a short cache TTL and a conservative, explicit fallback value baked into the SDK for when the cache is stale or fetch fails, rather than a pure binary choice between client-side and server-side. The common mistake is an SDK that silently serves an arbitrarily-old cached value with no TTL or fallback logic at all, which quietly turns "intermittent connectivity" into "unpredictable flag state," exactly the failure mode a well-designed flag system exists to prevent.

Direct answer
A canary deployment ships a new version to a small slice of traffic first, watches it closely against the stable version, and only widens exposure if it looks healthy; if it doesn't, you pull the plug on a small fraction of users instead of everyone.

Structured elaboration

1. Initial slice: route a small percentage of traffic, often 1-5%, to the new version while the rest continues on the stable version.
2. Observe: compare metrics between the canary and the stable baseline over the SAME time window, not the canary against yesterday's numbers, since traffic patterns shift by time of day.
3. Decide: if the canary's metrics stay within an acceptable band of the baseline for long enough, promote to a larger percentage; if they degrade, roll back the canary slice.
4. Ramp: repeat at increasing percentages (for example 5% -> 25% -> 100%) rather than jumping straight to full traffic, since a problem that only shows up under real production load or a particular traffic mix might not surface at 1%.
5. Promote or rollback trigger: could be a manual decision from a dashboard, or automated based on a metric threshold; either way it needs an explicit, pre-agreed criterion, not "it felt fine."

Worked example
A checkout service canaries a payment-processing change at 2% of traffic for 30 minutes. Error rate on the canary stays at 0.15% versus the stable version's 0.12%, well within the agreed 0.5% absolute-difference tolerance, so the team promotes to 25% for another 30 minutes, then to 100%.

Trade-offs and pitfalls
Canary buys you a much smaller blast radius than a straight rollout, but it's slower to reach full deployment and needs enough traffic volume for the canary slice to be statistically meaningful; a low-traffic service at 1% might only get a handful of requests, which isn't enough to detect a real but modest regression. The common mistake is treating a clean canary window as proof of correctness rather than as reduced risk: rare edge cases and slow-building problems (a memory leak, a cache-warming issue) can still slip through a short canary window.

Direct answer
This is the entry-level version of the deployment behavioral question: what was your role, how did you prepare, what surprised you, and what was the measurable outcome, even without a dramatic rollback story attached.

Structured elaboration

• Role: were you the one deploying, reviewing, on-call for it, or supporting? Be specific rather than vague about your actual involvement.
• Preparation: what did you do before shipping (tests written, a runbook checked, a rollback plan confirmed, a smaller-than-usual rollout percentage chosen because it was a first-time change)?
• A surprise, even a small one: interviews aren't looking for a disaster; a benign surprise (a metric moved differently than expected, a dependency behaved unexpectedly) still shows you were paying attention rather than deploying and walking away.
• Measurable outcome: a number if you have one (adoption rate, performance change, error rate before/after), or a concrete qualitative outcome if not.

Worked example
"I deployed a caching layer change for a read-heavy endpoint. I prepared by running the change through our staging load test first and setting up a dashboard specifically for the metrics I expected to move, latency and cache-hit rate, before shipping. The surprise was that cache-hit rate improved less than modeled, about 15 points instead of the 30 I'd projected, because a chunk of traffic had more request-parameter variability than our test data captured. The outcome was still a real 15-point improvement and a genuinely useful lesson about how our synthetic test traffic didn't reflect production request diversity, which changed how we built test fixtures afterward."

Trade-offs and pitfalls
The weakest version of this answer is generic ("it went well, no issues") with no specificity, which gives the interviewer nothing to probe and reads as either inexperience or a lack of real engagement with the deploy. Even a smooth, uneventful deployment has SOMETHING specific worth naming: a metric you watched, a decision you made about rollout size, a thing you learned.

Direct answer
Blue-green, canary, and rolling all reduce the risk of a bad release, but through different mechanisms: blue-green switches ALL traffic at once between two full environments, canary exposes a small SLICE of traffic to the new version before widening it, and rolling replaces instances gradually IN PLACE. A plain recreate deployment, by contrast, tears down the old version entirely before starting the new one, accepting downtime in exchange for simplicity.

Structured elaboration

Rollback complexity nuance: blue-green's rollback is fastest because the old environment never stopped running; canary and rolling both have to actively redeploy or re-route, which takes real time even if it's automated; recreate's "rollback" is simple mechanically but means accepting a second period of downtime.

Stateful services: all three of blue-green/canary/rolling get significantly harder with state (a database, in-memory session data, local disk), because you can't just duplicate or partially expose the data layer the way you can stateless compute; the deployment strategy for the STATELESS layer often decouples from a separate, more careful strategy for the DATA layer.

Worked example
A stateless API fronting a shared database: canary is a strong default, since it limits blast radius on the code change while the shared database (which doesn't get canaried the same way) stays constant underneath. Blue-green would be a better fit if the team's top priority is minimizing time-to-rollback over minimizing blast radius, since flipping back to the old environment is close to instant.

Trade-offs and pitfalls
There's no universally "best" strategy: the choice trades off blast radius, rollback speed, infrastructure cost, and operational complexity, and the right answer depends on which of those the specific service and change profile cares about most. A common mistake is picking a strategy based on what's trendy (everyone reaches for canary) rather than what the actual risk profile of the change calls for; a low-risk config change might not need any of this ceremony at all.