Safe Deployment and Rollback Strategies Questions

Direct answer
Blast radius is how much of your system, and how many users, are exposed to a bad deployment before you can stop it. Reducing it means never letting a single change reach 100% of traffic or 100% of your infrastructure in one step: you deploy to a small slice first, isolate that slice from the rest, and give yourself controls that can cut it off fast.

Structured elaboration
Techniques, roughly cheapest-to-hardest:

• Small-batch / percentage rollouts: canary a change to 1-5% of traffic or instances before going wider, so a bug affects a small fraction of users instead of everyone.
• Resource isolation: run the new version in separate compute (a distinct pod set, node pool, or availability zone) so a resource-exhaustion bug in the new version can't starve the old version's capacity too.
• Traffic controls: circuit breakers that stop routing to a demonstrably unhealthy instance, and rate limiters that cap how much load any single new component can absorb before it's proven stable.
• Region/cell isolation: for a global service, containing a rollout to one region or one "cell" of a sharded architecture means a bad release can't take down every region at once.
• Feature flags: decoupling "deployed" from "exposed" means you can turn a specific feature off instantly without a full redeploy, which is a much smaller and faster blast-radius-reduction lever than rolling back code.

For a monolith specifically, blast radius reduction is harder because there's no natural unit smaller than "the whole app": the levers become instance-level canarying (a subset of instances behind the load balancer run the new build) and feature flags around risky code paths, since you can't isolate one internal module's resource usage the way you can with a separate microservice.

Worked example
A change to a recommendation algorithm rolled out to 2% of traffic in one region first. A latency regression showed up only under that region's specific traffic mix (a caching quirk tied to timezone-driven request patterns); because it was contained to 2% of one region, the fix-and-redeploy cycle affected a small, recoverable slice of users instead of the whole global user base.

Trade-offs and pitfalls
More blast-radius controls mean more operational complexity and slower time-to-full-rollout, so teams calibrate the aggressiveness of containment to the risk of the change: a config tweak might skip straight to 100%, while a payment-logic change might go through five separate stages. The pitfall is applying the same heavy process to every change regardless of risk, which erodes the very safety discipline it's meant to protect by making people route around it under deadline pressure.

Direct answer
Rolling back a Kubernetes deployment with immutable image tags is kubectl rollout undo, which redeploys the previous ReplicaSet's pod template from Kubernetes' retained revision history; verification means confirming the rollout actually completed and the resulting pods are healthy, not just that the command returned success.

Structured elaboration and worked example

# 1. Confirm the current rollout history and identify the target revision (optional, but good practice)
kubectl rollout history deployment/checkout-api

# 2. Trigger the rollback to the immediately previous revision
kubectl rollout undo deployment/checkout-api

# 3. Watch it complete (this blocks until the rollout finishes or times out)
kubectl rollout status deployment/checkout-api --timeout=120s

# 4. Confirm the pods are actually on the expected previous image tag
kubectl get pods -l app=checkout-api -o jsonpath='{.items[*].spec.containers[*].image}'

# 5. Confirm readiness: all pods Running and Ready, not just Running
kubectl get pods -l app=checkout-api

# 6. Application-level health confirmation beyond kubectl's view of pod state
curl -sf https://checkout-api.internal/healthz

Why each verification step matters: rollout status blocking until completion (rather than assuming success the instant undo returns) catches a rollback that's stuck (for example, the previous image was garbage-collected from the registry and can't be pulled, which undo will happily accept as a command but which then fails to actually schedule). Checking the ACTUAL image tag on running pods, not just trusting the command succeeded, catches a mismatch between what you intended and what's actually running. The final application-level health check matters because Kubernetes' own view (Running, Ready) only confirms the CONTAINER started and passed its readiness probe, not that the application is functioning correctly for real traffic, the same distinction between "process is up" and "service actually works" that motivates smoke testing generally.

Trade-offs and pitfalls
kubectl rollout undo only has as many previous revisions to roll back to as revisionHistoryLimit allows (commonly 10, but sometimes reduced for resource reasons), so a rollback several versions back may fail if that history's been pruned, which is worth checking with rollout history before assuming undo will reach the version you actually want. The most common mistake is treating the undo command's successful return as confirmation the rollback WORKED, rather than explicitly waiting on rollout status and independently verifying pod health, since a rollback that's technically "issued" but stuck (image pull failure, resource constraints preventing new pods from scheduling) looks identical to a successful one until you check.

Direct answer
A rolling update replaces old-version instances with new-version ones gradually, a few at a time, so the service stays available with a mix of old and new versions running simultaneously during the transition. A recreate deployment, by contrast, terminates ALL old instances first and only then starts the new ones, which means a period of full downtime but avoids ever running mixed versions.

Structured elaboration
For a stateless microservice in Kubernetes, a rolling update:

1. Kubernetes creates a batch of new-version pods (controlled by maxSurge, how many extra pods above the target replica count are allowed).
2. Waits for those new pods to pass their readiness probe before routing traffic to them.
3. Terminates an equivalent batch of old-version pods (controlled by maxUnavailable, how many pods can be down at once).
4. Repeats until all pods are on the new version.

Common failure modes to watch for during a rollout:

• Readiness probe misconfigured too loosely: traffic gets routed to a pod that's technically "ready" but not actually able to serve correctly yet (cache not warmed, connection pool not established).
• Version skew during the mixed-version window: old and new pods running simultaneously both talk to the same downstream dependencies (shared database, shared cache), so if the new version isn't backward-compatible with what the old version expects, you get intermittent failures purely from which version happened to handle a given request.
• Resource exhaustion from surge: if maxSurge allows too many extra pods at once relative to available cluster capacity, new pods can fail to schedule, stalling the rollout partway.
• A bad new version rolling out gradually still affects a growing fraction of traffic before anyone notices, unlike blue-green where the bad version is fully isolated until an explicit cutover.

Worked example
A 20-replica deployment with maxSurge: 25% and maxUnavailable: 25% creates up to 5 extra pods (25 total temporarily) while taking down up to 5 old pods at a time, cycling through until all 20 are on the new version. If the new version has a subtle bug that only manifests under a specific downstream response, roughly a quarter of traffic is exposed to it at any point mid-rollout, growing toward 100% as the rollout proceeds, unless something halts it.

Trade-offs and pitfalls
Rolling update avoids downtime and extra infrastructure cost (no duplicate fleet, unlike blue-green) but accepts a mixed-version window where compatibility between old and new has to hold, and it doesn't isolate a bad release the way canary or blue-green does; it just gradually replaces capacity regardless of whether the new version is actually healthy, UNLESS combined with a readiness-probe-based or metrics-based halt condition.

Direct answer
This is the entry-level version of the deployment behavioral question: what was your role, how did you prepare, what surprised you, and what was the measurable outcome, even without a dramatic rollback story attached.

Structured elaboration

• Role: were you the one deploying, reviewing, on-call for it, or supporting? Be specific rather than vague about your actual involvement.
• Preparation: what did you do before shipping (tests written, a runbook checked, a rollback plan confirmed, a smaller-than-usual rollout percentage chosen because it was a first-time change)?
• A surprise, even a small one: interviews aren't looking for a disaster; a benign surprise (a metric moved differently than expected, a dependency behaved unexpectedly) still shows you were paying attention rather than deploying and walking away.
• Measurable outcome: a number if you have one (adoption rate, performance change, error rate before/after), or a concrete qualitative outcome if not.

Worked example
"I deployed a caching layer change for a read-heavy endpoint. I prepared by running the change through our staging load test first and setting up a dashboard specifically for the metrics I expected to move, latency and cache-hit rate, before shipping. The surprise was that cache-hit rate improved less than modeled, about 15 points instead of the 30 I'd projected, because a chunk of traffic had more request-parameter variability than our test data captured. The outcome was still a real 15-point improvement and a genuinely useful lesson about how our synthetic test traffic didn't reflect production request diversity, which changed how we built test fixtures afterward."

Trade-offs and pitfalls
The weakest version of this answer is generic ("it went well, no issues") with no specificity, which gives the interviewer nothing to probe and reads as either inexperience or a lack of real engagement with the deploy. Even a smooth, uneventful deployment has SOMETHING specific worth naming: a metric you watched, a decision you made about rollout size, a thing you learned.

Direct answer
A Helm hook backed by a Kubernetes Job can run a smoke test right after a release and, because Helm treats a failed hook Job as a failed release, automatically marks that release FAILED, which is what makes it useful as a real gate rather than just a post-deploy check someone has to notice manually.

Structured elaboration and worked example (YAML validated)

apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-api-smoke-test
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 120
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke-test
          image: registry.example.com/checkout-api-smoketests:1.42.0
          command: ["python3", "smoke_test.py", "--target", "http://checkout-api:8080"]

I parsed this manifest with a YAML validator to confirm it's structurally correct before including it.

How Helm handles rollback in this scenario: post-install/post-upgrade hooks run AFTER the main release resources are created, so the application is already deployed by the time this Job runs; if the Job fails (backoffLimit: 0 means no retries, a real failure fails fast rather than masking a transient blip as eventual success), Helm marks the release as failed in its own release history, which is what a downstream CI/CD step should check (helm status or the exit code of helm upgrade itself, since Helm propagates hook failure as a non-zero exit) to decide whether to trigger a rollback (helm rollback) automatically.

Keeping the hook idempotent: helm.sh/hook-delete-policy: before-hook-creation deletes any leftover Job from a PREVIOUS run before creating a new one, so re-running the same release (a retry, or Helm's own internal retry behavior) doesn't collide with a stale Job object still sitting around from before; hook-succeeded additionally cleans up the Job after a successful run, keeping the cluster tidy. activeDeadlineSeconds: 120 bounds how long the smoke test is allowed to hang before Kubernetes itself kills it and counts it as a failure, preventing a stuck smoke test from silently blocking the release indefinitely.

Trade-offs and pitfalls
Because this hook runs AFTER the application is already deployed and receiving traffic (a post-install/post-upgrade hook, not a pre-deploy gate), a failing smoke test here means users may have already been exposed to the bad release for the duration the hook takes to run and fail; for a lower-risk-tolerance service, pairing this with a pre-upgrade hook that tests against a not-yet-traffic-serving instance first (where feasible) reduces that exposure window further. The common mistake is a backoffLimit greater than 0 on a smoke-test Job, which retries a genuinely failing test multiple times before reporting failure, delaying the automatic-rollback decision for no real benefit.