Safe Deployment and Rollback Strategies Questions

Direct answer
Feature flags are runtime switches that let you turn a piece of behavior on or off without a redeploy, which decouples DEPLOYING code from RELEASING a feature to users. The common types are a release flag (temporarily gates a feature during rollout, meant to be removed once it's fully live), an experiment flag (drives an A/B test, removed once the experiment concludes), an operational flag (a longer-lived control, like a rate limiter toggle or a maintenance-mode switch), and a kill-switch (an emergency, instant off-switch for something risky).

Structured elaboration

• Release flags: owned by the engineer shipping the feature; short-lived by design, meant to be deleted once the feature is fully rolled out and stable.
• Experiment flags: owned jointly by product/data science and engineering; lifetime tied to the experiment's duration, removed once a winner is chosen.
• Operational flags: owned by the team operating the service; can be long-lived (a genuinely permanent operational lever), which is the ONE type where "long-lived" isn't automatically a problem.
• Kill-switches: owned by whoever's on-call; meant to be exercised rarely but tested regularly so it's trustworthy when actually needed.
• Common pitfalls as flags accumulate: flag debt (release flags that were never cleaned up after the feature fully shipped, cluttering the codebase with dead branches), flag sprawl (so many flags that nobody can reason about which combinations of flag states are even possible, let alone tested), and stale defaults (a flag's fallback value drifts out of sync with what's actually safe as the surrounding code evolves).

Worked example
A team ships a new checkout flow behind a release flag, ramps it from 5% to 100% of users over two weeks while watching conversion metrics, then deletes the flag and the old code path once it's fully rolled out and stable for a monitoring period. If that deletion step gets skipped (a common failure under deadline pressure to move to the next feature), six months later the codebase has a flag nobody remembers the purpose of, defaulting to a value nobody's sure is still correct.

Trade-offs and pitfalls
Flags buy real safety (instant, redeploy-free control over risky behavior) at the cost of code complexity: every flag is effectively a branch that has to be reasoned about and eventually tested in both states. The single most common failure mode in practice isn't misusing a flag while it's active, it's forgetting to remove it once its job is done, which is why healthy flag programs bake in an explicit cleanup step (an expiration date, a dashboard of stale flags, or a policy that blocks new flags until old ones are retired) rather than relying on developers to remember.

Direct answer
Automatic, SLO-triggered rollback is faster and removes human latency from the reaction time, but it can't exercise judgment about WHY a violation happened or whether reverting is actually the right response; a manual, human-in-the-loop gate is slower but can weigh context automation can't, which matters enormously in a domain like healthcare where the wrong automated action itself carries real risk and regulatory consequence.

Structured elaboration

• Speed of detection vs. speed of correct decision: automatic rollback reacts the instant a metric crosses a threshold, which is valuable when the threshold reliably indicates a genuine problem; but an ML system's "metric" (model accuracy on some proxy signal, prediction-distribution drift) is often a noisier, less directly interpretable signal than a simple system health check, so automating the ROLLBACK DECISION on a noisy proxy risks reacting to something that isn't actually the problem it appears to be.
• Risk of flapping: an ML model's output distribution can shift for benign reasons (a genuine change in the underlying population being scored, not a model regression), which an automated threshold can't easily distinguish from an actual regression, making false-positive automated rollbacks a real, recurring risk in ML systems specifically, more so than for typical application code.
• Regulatory reporting obligations: in a regulated domain, an automated action taken on a live model (disabling or reverting a clinical-decision-support model, for instance) may itself trigger a reporting or review obligation, meaning the "fast, automated" path isn't actually free of process overhead, it just moves that overhead to AFTER the action instead of before it.
• Patient safety trade-off, specifically: a model continuing to make bad predictions for the time it takes a human to review is a real patient-safety risk; but an automated system incorrectly reverting a model that was actually fine (based on a noisy proxy signal) also has a cost, potentially reverting to an older, genuinely worse-performing model, which is its own patient-safety risk in the opposite direction.

A hybrid policy: automate the PAUSE (stop routing new decisions to the model, falling back to a safe default or a human-reviewed process) on SLO violation, since pausing is lower-risk and more easily reversible than a full rollback; but require human confirmation before the FULL rollback/replacement decision, giving a clinician or ML-safety reviewer time to assess whether the violation reflects a genuine model problem or a benign population shift, while the pause itself already protects patients from continued exposure to a potentially-bad model during that review window.

Worked example
A diagnostic-support model's accuracy proxy metric drops below threshold: the automated system immediately pauses the model (falling back to the previous model or a manual-review workflow, whichever is the domain's established safe default) within minutes, protecting patients from continued exposure without requiring a human to react instantly. A clinical-ML-safety reviewer then investigates within an SLA-bound window (say, one hour) to determine whether this is a genuine regression (confirm full rollback) or a benign population shift (resume the model, with the pause itself now logged as part of the regulatory reporting trail either way).

Trade-offs and pitfalls
The hybrid design's real strength is separating a LOW-RISK, FAST, automatable action (pause) from a HIGHER-STAKES decision (full rollback/replacement) that benefits from human judgment, rather than treating "automatic" and "manual" as the only two options for the entire response. The common mistake is either fully automating the rollback decision itself (risking flapping and patient-safety-relevant false positives) or requiring full manual review even for the pause step (unnecessarily slowing the one action that's actually safe to automate).

Direct answer
When a newly deployed model shows a real regression, the rollback options range from fastest-but-broadest to slower-but-more-targeted: a feature-flag disable or hot-swap to the previous model version are usually the fastest and safest first moves, while aborting an in-progress canary or falling back to shadow mode are options specifically available if you hadn't yet fully committed the new model to live traffic.

Structured elaboration

• Feature-flag disable: if the new model sits behind a flag, flipping it off instantly routes back to the previous model or a fallback rule-based path, with no redeploy needed; this is usually the fastest option and the first one to reach for if it's available.
• Hot-swap to the previous model: if the serving infrastructure keeps the previous model version warm and loadable, switching the active model pointer is close to instant and doesn't require a full redeploy of the serving service, only a config/pointer change.
• Blue-green rollback: if the model was deployed via a blue-green pattern at the serving-infrastructure level, switching traffic back to the previous serving environment is as fast as any blue-green rollback, at the cost of the infrastructure overhead of having kept a full previous environment warm.
• Aborting a canary: if the regression is caught while the new model is still only receiving a slice of traffic (not yet fully promoted), simply stopping the canary and routing all traffic back to the previous model is both fast and low-risk, since most traffic was never exposed to the bad model in the first place.
• Falling back to shadowing: if you're not confident the previous model is the right target either (maybe you want to keep learning about the new model's behavior without exposing users to it), routing the new model to shadow mode (receiving a copy of traffic but not affecting real responses) while live traffic goes back to the previous model gets you both safety and continued visibility.

Time-to-recovery and validation, roughly fastest to slowest: feature-flag disable and canary-abort are typically seconds; hot-swap is seconds to low minutes depending on model-loading time; blue-green rollback is minutes; each needs post-rollback validation that the false-positive rate (or whatever the regressed metric was) has actually returned to baseline on the RESTORED model, not just an assumption that reverting fixes it, since the regression could in principle have a cause unrelated to the model itself (a data-pipeline issue feeding both versions, for example).

Worked example
A model regression discovered via a 7% false-positive increase, with the new model deployed behind a feature flag at 100% traffic (fully promoted, not mid-canary): flip the flag to route back to the previous model immediately (seconds), then validate the false-positive rate over the next several minutes of restored traffic actually returns to its prior baseline, confirming the rollback addressed the regression rather than coinciding with an unrelated fix.

Trade-offs and pitfalls
The biggest pitfall is treating "we rolled back" as automatically meaning "the problem is solved" without validating the metric actually recovers; if the false-positive spike was caused by an upstream data issue affecting BOTH models, rolling back the model alone wouldn't fix it, and validation is what catches that distinction. The other common mistake is not having ANY of these fast paths (no flag, no hot-swap capability, no blue-green) available for a model deployment, forcing a full redeploy as the only option even for the highest-stakes, fastest-moving incidents.

Direct answer
A canary deployment ships a new version to a small slice of traffic first, watches it closely against the stable version, and only widens exposure if it looks healthy; if it doesn't, you pull the plug on a small fraction of users instead of everyone.

Structured elaboration

1. Initial slice: route a small percentage of traffic, often 1-5%, to the new version while the rest continues on the stable version.
2. Observe: compare metrics between the canary and the stable baseline over the SAME time window, not the canary against yesterday's numbers, since traffic patterns shift by time of day.
3. Decide: if the canary's metrics stay within an acceptable band of the baseline for long enough, promote to a larger percentage; if they degrade, roll back the canary slice.
4. Ramp: repeat at increasing percentages (for example 5% -> 25% -> 100%) rather than jumping straight to full traffic, since a problem that only shows up under real production load or a particular traffic mix might not surface at 1%.
5. Promote or rollback trigger: could be a manual decision from a dashboard, or automated based on a metric threshold; either way it needs an explicit, pre-agreed criterion, not "it felt fine."

Worked example
A checkout service canaries a payment-processing change at 2% of traffic for 30 minutes. Error rate on the canary stays at 0.15% versus the stable version's 0.12%, well within the agreed 0.5% absolute-difference tolerance, so the team promotes to 25% for another 30 minutes, then to 100%.

Trade-offs and pitfalls
Canary buys you a much smaller blast radius than a straight rollout, but it's slower to reach full deployment and needs enough traffic volume for the canary slice to be statistically meaningful; a low-traffic service at 1% might only get a handful of requests, which isn't enough to detect a real but modest regression. The common mistake is treating a clean canary window as proof of correctness rather than as reduced risk: rare edge cases and slow-building problems (a memory leak, a cache-warming issue) can still slip through a short canary window.

Direct answer
The recreate deployment strategy tears down every instance of the old version before starting any instance of the new one, accepting a period of full downtime in exchange for the simplest possible mental model: at any given moment, either all-old or all-new is running, never a mix.

Structured elaboration
Teams still choose recreate, despite the downtime, when:

• Mixed-version compatibility is genuinely hard or unsafe to reason about: if old and new code can't safely coexist even briefly (a breaking, non-backward-compatible change with no feasible way to make it compatible for a transition window), recreate avoids ever creating that mixed-version state at all, at the cost of downtime instead.
• The service can tolerate a maintenance window: an internal tool, a batch-processing system, or a service with well-understood low-traffic periods can schedule the downtime somewhere it genuinely doesn't matter much, making the simplicity worth the (planned, contained) cost.
• Resource constraints: environments without spare capacity for even a small surge (an embedded system, a resource-constrained edge deployment, a small on-premises cluster with no slack) may not have room to run old and new simultaneously even briefly, making recreate the only option that fits the available resources.
• Simplicity as a deliberate choice for a low-stakes service: not every service justifies the operational complexity of a rolling update or canary; a low-traffic internal dashboard might reasonably just accept a 30-second restart rather than investing in zero-downtime tooling for something that doesn't need it.

Worked example
A nightly batch-processing job that only runs during a defined off-peak window: recreate is not just acceptable but arguably the RIGHT choice, since there's no live traffic to protect during that window anyway, and the operational simplicity (no need to reason about version-skew compatibility, no readiness-probe tuning) reduces real complexity for no actual downside given the service's usage pattern.

Trade-offs and pitfalls
The common mistake is defaulting to recreate purely out of inertia or unfamiliarity with rolling/canary patterns, on a service that actually DOES have meaningful live traffic and would benefit from a more careful strategy; recreate should be a deliberate choice matched to a specific service's tolerance for downtime, not the path of least resistance for every service regardless of its actual traffic pattern.