Safe Deployment and Rollback Strategies Questions

Direct answer
A canary deployment ships a new version to a small slice of traffic first, watches it closely against the stable version, and only widens exposure if it looks healthy; if it doesn't, you pull the plug on a small fraction of users instead of everyone.

Structured elaboration

1. Initial slice: route a small percentage of traffic, often 1-5%, to the new version while the rest continues on the stable version.
2. Observe: compare metrics between the canary and the stable baseline over the SAME time window, not the canary against yesterday's numbers, since traffic patterns shift by time of day.
3. Decide: if the canary's metrics stay within an acceptable band of the baseline for long enough, promote to a larger percentage; if they degrade, roll back the canary slice.
4. Ramp: repeat at increasing percentages (for example 5% -> 25% -> 100%) rather than jumping straight to full traffic, since a problem that only shows up under real production load or a particular traffic mix might not surface at 1%.
5. Promote or rollback trigger: could be a manual decision from a dashboard, or automated based on a metric threshold; either way it needs an explicit, pre-agreed criterion, not "it felt fine."

Worked example
A checkout service canaries a payment-processing change at 2% of traffic for 30 minutes. Error rate on the canary stays at 0.15% versus the stable version's 0.12%, well within the agreed 0.5% absolute-difference tolerance, so the team promotes to 25% for another 30 minutes, then to 100%.

Trade-offs and pitfalls
Canary buys you a much smaller blast radius than a straight rollout, but it's slower to reach full deployment and needs enough traffic volume for the canary slice to be statistically meaningful; a low-traffic service at 1% might only get a handful of requests, which isn't enough to detect a real but modest regression. The common mistake is treating a clean canary window as proof of correctness rather than as reduced risk: rare edge cases and slow-building problems (a memory leak, a cache-warming issue) can still slip through a short canary window.

Direct answer
An error-budget deployment gate reads the service's SLO configuration, computes how much of the allowed error budget has already been consumed over the measurement window, and blocks the deploy if the remaining budget drops below a threshold, typically living as a required check right before the production-promotion stage of the pipeline.

Structured elaboration
The gate needs: (1) the SLO's target (say 99.9% availability), from which the allowed error rate is 1 - target; (2) the OBSERVED error rate over the rolling window (30 days is common, though shorter windows react faster to recent degradation); (3) a computation of remaining budget as a percentage of the ALLOWED budget, not of total traffic, since "5% of allowed budget remaining" is a very different, more urgent statement than "5% error rate"; (4) a threshold below which the gate blocks (10% remaining is a common conservative choice).

Worked example (executed)

from dataclasses import dataclass

@dataclass
class SLOConfig:
    target_availability: float
    window_days: int = 30

def compute_remaining_budget_pct(slo: SLOConfig, observed_error_rate: float) -> float:
    allowed_error_rate = 1 - slo.target_availability
    remaining_fraction = 1 - (observed_error_rate / allowed_error_rate)
    return remaining_fraction * 100

def deployment_gate(slo: SLOConfig, observed_error_rate: float, min_remaining_pct: float = 10.0):
    remaining_pct = compute_remaining_budget_pct(slo, observed_error_rate)
    return remaining_pct >= min_remaining_pct, remaining_pct

slo = SLOConfig(target_availability=0.999)  # allowed error rate = 0.1%

Run against three cases: a healthy service at 0.02% observed error returns (True, 80.0), 80% of budget still available, deploy proceeds. A service that's burned most of its budget, observed at 0.095%, returns (False, 5.0), only 5% left, below the 10% floor, deploy blocked. A service that's blown past its budget entirely, observed at 0.15% against a 0.1% allowance, returns (False, -50.0), a negative number correctly signaling the budget is already exhausted rather than clamping at zero, which matters because "50% over budget" and "exactly at budget" should trigger differently urgent responses even though both block the deploy.

Pipeline placement
This gate sits as a required check immediately before the "promote to production" step, after build/test/staging have already passed, since it's answering "should THIS release happen right now," not "is the code correct." It should have an explicit bypass path for emergency fixes (a rollback or a critical hotfix that's REDUCING risk, not adding it), gated by an approval rather than silently exempt, so the override is visible and auditable.

Trade-offs and pitfalls
A 30-day window reacts slowly to a service that's degrading right now; a shorter window reacts faster but is noisier and can block deploys over a transient blip that's already resolved. The common pitfall is computing remaining budget as a percentage of TOTAL traffic instead of the ALLOWED budget, which massively understates how urgent the situation is: 0.08% error rate sounds fine in isolation, but against a 0.1% allowance it's already 80% of the budget gone.

Direct answer
Blast radius is how much of your system, and how many users, are exposed to a bad deployment before you can stop it. Reducing it means never letting a single change reach 100% of traffic or 100% of your infrastructure in one step: you deploy to a small slice first, isolate that slice from the rest, and give yourself controls that can cut it off fast.

Structured elaboration
Techniques, roughly cheapest-to-hardest:

• Small-batch / percentage rollouts: canary a change to 1-5% of traffic or instances before going wider, so a bug affects a small fraction of users instead of everyone.
• Resource isolation: run the new version in separate compute (a distinct pod set, node pool, or availability zone) so a resource-exhaustion bug in the new version can't starve the old version's capacity too.
• Traffic controls: circuit breakers that stop routing to a demonstrably unhealthy instance, and rate limiters that cap how much load any single new component can absorb before it's proven stable.
• Region/cell isolation: for a global service, containing a rollout to one region or one "cell" of a sharded architecture means a bad release can't take down every region at once.
• Feature flags: decoupling "deployed" from "exposed" means you can turn a specific feature off instantly without a full redeploy, which is a much smaller and faster blast-radius-reduction lever than rolling back code.

For a monolith specifically, blast radius reduction is harder because there's no natural unit smaller than "the whole app": the levers become instance-level canarying (a subset of instances behind the load balancer run the new build) and feature flags around risky code paths, since you can't isolate one internal module's resource usage the way you can with a separate microservice.

Worked example
A change to a recommendation algorithm rolled out to 2% of traffic in one region first. A latency regression showed up only under that region's specific traffic mix (a caching quirk tied to timezone-driven request patterns); because it was contained to 2% of one region, the fix-and-redeploy cycle affected a small, recoverable slice of users instead of the whole global user base.

Trade-offs and pitfalls
More blast-radius controls mean more operational complexity and slower time-to-full-rollout, so teams calibrate the aggressiveness of containment to the risk of the change: a config tweak might skip straight to 100%, while a payment-logic change might go through five separate stages. The pitfall is applying the same heavy process to every change regardless of risk, which erodes the very safety discipline it's meant to protect by making people route around it under deadline pressure.

Direct answer
A post-deploy health-poll script watches the new version for a fixed window right after it starts receiving traffic, aggregating success rate and latency, and explicitly marks the deployment failed (triggering whatever the pipeline's next step is, typically an automatic rollback) if either crosses a threshold, rather than assuming silence means success.

Structured elaboration and worked example (logic verified)

import time
import requests

def poll_health(url: str, duration_seconds: int = 300, interval_seconds: int = 5,
                 success_rate_threshold: float = 0.99, latency_threshold_ms: float = 500):
    # Polls a health endpoint for `duration_seconds`, aggregating results.
    # Returns (passed: bool, summary: dict).
    start = time.monotonic()
    successes = 0
    total = 0
    latencies = []

    while time.monotonic() - start < duration_seconds:
        req_start = time.monotonic()
        try:
            resp = requests.get(url, timeout=5)
            elapsed_ms = (time.monotonic() - req_start) * 1000
            latencies.append(elapsed_ms)
            total += 1
            if resp.status_code == 200:
                successes += 1
        except requests.RequestException:
            total += 1
            latencies.append(latency_threshold_ms * 10)  # count a hard failure as very slow, not silently dropped
        time.sleep(interval_seconds)

    if total == 0:
        return False, {"reason": "no requests completed, cannot assess health"}

    success_rate = successes / total
    avg_latency = sum(latencies) / len(latencies)
    passed = success_rate >= success_rate_threshold and avg_latency <= latency_threshold_ms
    return passed, {"success_rate": success_rate, "avg_latency_ms": avg_latency, "requests": total}

Key structural decisions: a network exception is counted as a failed, slow request rather than silently skipped, since a connection timeout is itself a strong negative signal that shouldn't be excluded from the aggregate just because it didn't return an HTTP status code at all. Zero completed requests is treated as an explicit failure with a clear reason, never silently passed, since "we don't have enough data" should never be mistaken for "it's healthy."

Trade-offs and pitfalls
A fixed polling interval (5 seconds here) trades responsiveness for load on the health endpoint; a service under real stress from a bad deploy doesn't need to be hammered by an aggressive health-check loop on top of everything else. The retry/timeout handling matters more than it looks: a script that raises an unhandled exception on the FIRST network hiccup, rather than counting it as a data point and continuing, produces a false "script crashed" result instead of the actually useful "service is unhealthy" signal the pipeline needs to act on.

Direct answer
This is a judgment-under-pressure story, not a pure technical one: the interviewer wants to see how you weigh delivery speed against safety in a real, specific situation, including what mitigations you reached for and what you'd do differently with hindsight.

Structured elaboration

• Set up the tension honestly: what was the actual pressure (a deadline, a competitor move, an executive ask) and what was the actual risk you were weighing against it?
• Name the mitigations you used: a feature flag so the risky part could be turned off instantly, a canary at a smaller-than-usual percentage, a staged rollback plan (an explicit ramp with a defined rollback checkpoint at each stage, so a bad sign at 5% never reaches the next stage instead of discovering the problem only after 100%), an extra pair of eyes on the specific risky code path, and a rollback plan written down BEFORE shipping rather than improvised after.
• Be honest about the outcome: a good answer doesn't require the decision to have been perfect; it requires the reasoning to have been sound given what was known at the time, and ideally an honest account of what you learned even if things went fine.
• If you don't have a direct example: present the decision framework you'd actually use: what factors would tip you toward speed (low blast radius, easy rollback, low-stakes feature) versus toward safety (payment/auth-adjacent, hard-to-reverse, high-traffic).

Worked example
"We had a hard external deadline (a partner integration going live) that pushed us to ship a change to our API rate-limiting logic faster than our normal review cycle. I pushed to keep the change behind a flag defaulting OFF for everyone except the specific partner's traffic, so the blast radius if something was wrong was contained to one integration rather than global. On top of the flag, we staged the ramp explicitly: the partner's traffic first, then our next three largest customers a day later once nothing looked off, then everyone else, with an agreed rollback checkpoint (a defined error-rate band) at each stage rather than one all-or-nothing cutover. We also wrote the rollback plan (just flip the flag) before shipping, not after. It turned out fine, but the flag and staged ramp meant that if it hadn't, the fix would have taken seconds and affected a small, known slice of traffic instead of a full redeploy against everyone at once."

Trade-offs and pitfalls
A common weak answer treats this as either "we always prioritize safety" (which reads as inexperienced with real deadline pressure) or "we shipped fast and got lucky" (which reads as reckless); the strongest answers show a considered trade-off with a concrete mitigation, such as a flag, a canary, or a staged rollback, that reduced the actual risk of the fast path, rather than just accepting the risk unmitigated.