Safe Deployment and Rollback Strategies Questions

Direct answer
This is a judgment-under-pressure story, not a pure technical one: the interviewer wants to see how you weigh delivery speed against safety in a real, specific situation, including what mitigations you reached for and what you'd do differently with hindsight.

Structured elaboration

• Set up the tension honestly: what was the actual pressure (a deadline, a competitor move, an executive ask) and what was the actual risk you were weighing against it?
• Name the mitigations you used: a feature flag so the risky part could be turned off instantly, a canary at a smaller-than-usual percentage, a staged rollback plan (an explicit ramp with a defined rollback checkpoint at each stage, so a bad sign at 5% never reaches the next stage instead of discovering the problem only after 100%), an extra pair of eyes on the specific risky code path, and a rollback plan written down BEFORE shipping rather than improvised after.
• Be honest about the outcome: a good answer doesn't require the decision to have been perfect; it requires the reasoning to have been sound given what was known at the time, and ideally an honest account of what you learned even if things went fine.
• If you don't have a direct example: present the decision framework you'd actually use: what factors would tip you toward speed (low blast radius, easy rollback, low-stakes feature) versus toward safety (payment/auth-adjacent, hard-to-reverse, high-traffic).

Worked example
"We had a hard external deadline (a partner integration going live) that pushed us to ship a change to our API rate-limiting logic faster than our normal review cycle. I pushed to keep the change behind a flag defaulting OFF for everyone except the specific partner's traffic, so the blast radius if something was wrong was contained to one integration rather than global. On top of the flag, we staged the ramp explicitly: the partner's traffic first, then our next three largest customers a day later once nothing looked off, then everyone else, with an agreed rollback checkpoint (a defined error-rate band) at each stage rather than one all-or-nothing cutover. We also wrote the rollback plan (just flip the flag) before shipping, not after. It turned out fine, but the flag and staged ramp meant that if it hadn't, the fix would have taken seconds and affected a small, known slice of traffic instead of a full redeploy against everyone at once."

Trade-offs and pitfalls
A common weak answer treats this as either "we always prioritize safety" (which reads as inexperienced with real deadline pressure) or "we shipped fast and got lucky" (which reads as reckless); the strongest answers show a considered trade-off with a concrete mitigation, such as a flag, a canary, or a staged rollback, that reduced the actual risk of the fast path, rather than just accepting the risk unmitigated.

Direct answer
Server-side evaluation is generally safer for a rollout because the server can change a flag's value instantly and consistently for every client, while client-side evaluation depends on each device having up-to-date flag configuration, which is exactly what breaks down under intermittent connectivity.

Structured elaboration

• Client-side evaluation: the app itself decides whether a feature is on, usually based on a flag configuration it fetched and cached at some point. Faster to evaluate at request time (no network round-trip needed) and works fully offline, but a device on a stale cache keeps using an OLD flag value until it next successfully syncs, which for an intermittently-connected mobile app could be minutes to hours out of date.
• Server-side evaluation: the server decides on each request. Always reflects the current flag state instantly and consistently, but requires a live connection for every decision that depends on the flag, which is a problem for a mobile app trying to work offline or under a flaky connection.
• Default behavior when a flag can't be fetched: this is the crux of the safety question. The system needs an explicit, deliberately chosen default for "no data available," and that default should almost always be the SAFE, conservative behavior (feature OFF, or the old known-good code path), never "assume the last cached value is still correct" for anything risk-sensitive, since an unreachable device might be running a stale cache for an unknown, possibly long, period.

Worked example
A payment-related feature flag on a mobile app with client-side evaluation: the SDK is configured so that if it can't reach the flag service within a short timeout, it falls back to a hardcoded, safe default (feature OFF) rather than serving whatever was last cached, which might be hours or days stale if the device has been offline. This costs some feature-availability under poor connectivity (some users see the old behavior more often than strictly necessary) in exchange for never risking a stale, potentially-unsafe flag state driving payment logic.

Trade-offs and pitfalls
A hybrid is common in practice: client-side evaluation for speed and offline resilience, paired with a short cache TTL and a conservative, explicit fallback value baked into the SDK for when the cache is stale or fetch fails, rather than a pure binary choice between client-side and server-side. The common mistake is an SDK that silently serves an arbitrarily-old cached value with no TTL or fallback logic at all, which quietly turns "intermittent connectivity" into "unpredictable flag state," exactly the failure mode a well-designed flag system exists to prevent.

Direct answer
Blue-green, canary, and rolling all reduce the risk of a bad release, but through different mechanisms: blue-green switches ALL traffic at once between two full environments, canary exposes a small SLICE of traffic to the new version before widening it, and rolling replaces instances gradually IN PLACE. A plain recreate deployment, by contrast, tears down the old version entirely before starting the new one, accepting downtime in exchange for simplicity.

Structured elaboration

Rollback complexity nuance: blue-green's rollback is fastest because the old environment never stopped running; canary and rolling both have to actively redeploy or re-route, which takes real time even if it's automated; recreate's "rollback" is simple mechanically but means accepting a second period of downtime.

Stateful services: all three of blue-green/canary/rolling get significantly harder with state (a database, in-memory session data, local disk), because you can't just duplicate or partially expose the data layer the way you can stateless compute; the deployment strategy for the STATELESS layer often decouples from a separate, more careful strategy for the DATA layer.

Worked example
A stateless API fronting a shared database: canary is a strong default, since it limits blast radius on the code change while the shared database (which doesn't get canaried the same way) stays constant underneath. Blue-green would be a better fit if the team's top priority is minimizing time-to-rollback over minimizing blast radius, since flipping back to the old environment is close to instant.

Trade-offs and pitfalls
There's no universally "best" strategy: the choice trades off blast radius, rollback speed, infrastructure cost, and operational complexity, and the right answer depends on which of those the specific service and change profile cares about most. A common mistake is picking a strategy based on what's trendy (everyone reaches for canary) rather than what the actual risk profile of the change calls for; a low-risk config change might not need any of this ceremony at all.

Direct answer
Feature flags are runtime switches that let you turn a piece of behavior on or off without a redeploy, which decouples DEPLOYING code from RELEASING a feature to users. The common types are a release flag (temporarily gates a feature during rollout, meant to be removed once it's fully live), an experiment flag (drives an A/B test, removed once the experiment concludes), an operational flag (a longer-lived control, like a rate limiter toggle or a maintenance-mode switch), and a kill-switch (an emergency, instant off-switch for something risky).

Structured elaboration

• Release flags: owned by the engineer shipping the feature; short-lived by design, meant to be deleted once the feature is fully rolled out and stable.
• Experiment flags: owned jointly by product/data science and engineering; lifetime tied to the experiment's duration, removed once a winner is chosen.
• Operational flags: owned by the team operating the service; can be long-lived (a genuinely permanent operational lever), which is the ONE type where "long-lived" isn't automatically a problem.
• Kill-switches: owned by whoever's on-call; meant to be exercised rarely but tested regularly so it's trustworthy when actually needed.
• Common pitfalls as flags accumulate: flag debt (release flags that were never cleaned up after the feature fully shipped, cluttering the codebase with dead branches), flag sprawl (so many flags that nobody can reason about which combinations of flag states are even possible, let alone tested), and stale defaults (a flag's fallback value drifts out of sync with what's actually safe as the surrounding code evolves).

Worked example
A team ships a new checkout flow behind a release flag, ramps it from 5% to 100% of users over two weeks while watching conversion metrics, then deletes the flag and the old code path once it's fully rolled out and stable for a monitoring period. If that deletion step gets skipped (a common failure under deadline pressure to move to the next feature), six months later the codebase has a flag nobody remembers the purpose of, defaulting to a value nobody's sure is still correct.

Trade-offs and pitfalls
Flags buy real safety (instant, redeploy-free control over risky behavior) at the cost of code complexity: every flag is effectively a branch that has to be reasoned about and eventually tested in both states. The single most common failure mode in practice isn't misusing a flag while it's active, it's forgetting to remove it once its job is done, which is why healthy flag programs bake in an explicit cleanup step (an expiration date, a dashboard of stale flags, or a policy that blocks new flags until old ones are retired) rather than relying on developers to remember.

Direct answer
A canary deployment ships a new version to a small slice of traffic first, watches it closely against the stable version, and only widens exposure if it looks healthy; if it doesn't, you pull the plug on a small fraction of users instead of everyone.

Structured elaboration

1. Initial slice: route a small percentage of traffic, often 1-5%, to the new version while the rest continues on the stable version.
2. Observe: compare metrics between the canary and the stable baseline over the SAME time window, not the canary against yesterday's numbers, since traffic patterns shift by time of day.
3. Decide: if the canary's metrics stay within an acceptable band of the baseline for long enough, promote to a larger percentage; if they degrade, roll back the canary slice.
4. Ramp: repeat at increasing percentages (for example 5% -> 25% -> 100%) rather than jumping straight to full traffic, since a problem that only shows up under real production load or a particular traffic mix might not surface at 1%.
5. Promote or rollback trigger: could be a manual decision from a dashboard, or automated based on a metric threshold; either way it needs an explicit, pre-agreed criterion, not "it felt fine."

Worked example
A checkout service canaries a payment-processing change at 2% of traffic for 30 minutes. Error rate on the canary stays at 0.15% versus the stable version's 0.12%, well within the agreed 0.5% absolute-difference tolerance, so the team promotes to 25% for another 30 minutes, then to 100%.

Trade-offs and pitfalls
Canary buys you a much smaller blast radius than a straight rollout, but it's slower to reach full deployment and needs enough traffic volume for the canary slice to be statistically meaningful; a low-traffic service at 1% might only get a handful of requests, which isn't enough to detect a real but modest regression. The common mistake is treating a clean canary window as proof of correctness rather than as reduced risk: rare edge cases and slow-building problems (a memory leak, a cache-warming issue) can still slip through a short canary window.