Safe Deployment and Rollback Strategies Questions

Direct answer
An annotated Kubernetes Deployment manifest for a stateless web service needs readiness and liveness probes to gate traffic and recovery correctly, resource requests/limits so the scheduler places it sensibly and it doesn't starve or get starved by neighbors, and a rollingUpdate strategy tuned to keep the service fully available throughout a rollout.

Structured elaboration and worked example (YAML validated)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 100
  selector:
    matchLabels:
      app: checkout-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%   # up to 25 of 100 pods down at once; keeps 75+ capacity live throughout
      maxSurge: 25%         # up to 25 extra pods above the 100 target while new pods come up
  template:
    metadata:
      labels:
        app: checkout-api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.42.0   # immutable, digest-pinnable tag, never ':latest'
          ports:
            - containerPort: 8080
            - containerPort: 9090
              name: metrics
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3

I parsed this manifest with a YAML validator to confirm structural correctness before including it here; the parser confirmed maxUnavailable: 25% and maxSurge: 25% and both probe blocks parse as valid structures.

Why each choice helps:

• maxUnavailable: 25% / maxSurge: 25%: balances rollout speed against never dropping below 75% capacity, a reasonable default for a service that can tolerate some capacity reduction but shouldn't lose too much at once.
• Separate readinessProbe and livenessProbe endpoints (/healthz/ready vs /healthz/live): readiness can legitimately fail transiently (a dependency is briefly unreachable) without the pod being killed; liveness should only fail for a genuinely stuck process, so conflating them risks unnecessary restarts under normal transient conditions.
• initialDelaySeconds: 5 (readiness) shorter than 15 (liveness): the app should be able to answer readiness soon after starting, but liveness gets a longer grace period so a slightly slow boot doesn't trigger a restart before the app has had a fair chance to come up.
• Explicit resources.requests and limits: requests inform the scheduler's placement decisions (and directly affect maxSurge's actual resource cost, since 25 extra pods at these requests is a real, calculable resource ask on the cluster); limits prevent one runaway pod from starving its node-mates.

Trade-offs and pitfalls
Setting maxSurge higher than the cluster actually has spare capacity for causes new pods to stall in Pending, silently stalling the whole rollout; this manifest's values are a reasonable default but should be validated against actual cluster headroom, not chosen in isolation. A common mistake is pointing both probes at the same endpoint with identical logic, which defeats the purpose of having two separate mechanisms with two different jobs (traffic gating vs. restart decision).

Direct answer

Imperative CI/CD's rollback model is "run the DEPLOY step again with the previous artifact," a forward-facing operation on the deployment TOOL; GitOps's rollback model is "make Git say what it said before," a backward-facing operation on the SOURCE OF TRUTH, and the deployment mechanism (the reconciler) then does exactly what it always does, reconcile the cluster toward whatever Git currently declares. This is a genuine model shift, not just a different command: in GitOps, there is no separate "rollback" code path at all, reverting IS just another declarative change, going through the identical PR-and-reconcile flow as any other change, including review.

Structured elaboration

How GitOps changes the rollback model. Imperative CI/CD typically has rollback as its OWN distinct mechanism (a "redeploy previous version" pipeline action, sometimes with different permissions or a different code path than a normal deploy). GitOps has NO separate rollback mechanism; a revert is structurally identical to any other change, a Git commit that changes the declared state, reviewed the same way, applied by the same reconciliation loop. This means rollback INHERITS every safety property (review, audit trail, policy-as-code evaluation) normal changes get, rather than needing those properties separately re-implemented for a distinct rollback code path.

Concrete steps to revert a bad deployment.

1. Identify the specific commit that introduced the bad change (using the same evidence-first approach any outage investigation follows).
2. git revert <bad-commit> (or, for a promotion-based flow, a new commit re-pinning the PREVIOUS artifact reference), creating a NEW commit undoing the change, never force-pushing over history.
3. This revert PR goes through the SAME review as any change (expedited under the emergency-change path if genuinely urgent, but still a real, recorded review, never skipped entirely).
4. On merge, the GitOps controller (Argo CD or Flux) detects the Git state has changed and reconciles automatically, no separate "trigger a rollback" action needed beyond the normal merge that any change would need.

Ensuring the cluster reconciles to the reverted state safely. Adapting the same rollback caution used for Terraform to Kubernetes: confirm the revert's computed diff (what the reconciler will actually change) BEFORE merging, the same plan-review discipline applies here to a GitOps revert too, since "just revert the commit" does not guarantee the resulting change is itself risk-free (a revert can, in principle, trigger its own significant change if enough time and other changes have passed since the original commit). Once merged, monitor the reconciliation completing successfully, not just assume merge equals resolved, a revert PR can itself partially fail to apply, exactly like any other change.

Worked example

A bad deployment of checkout-api (a broken config value) via Argo CD:

Imperative CI/CD equivalent (for contrast): trigger the CI/CD pipeline's "redeploy previous version" action, which re-runs the deploy step against the prior artifact; this is a DIFFERENT operation from a normal deploy, often with its own separate permissions and, depending on the pipeline's design, potentially LESS reviewed than a normal deploy would be (rollback is often treated as an emergency action that skips normal process).

GitOps:

git log --oneline apps/checkout-api/overlays/prod/   # identify the bad commit
git revert <bad-commit-sha>                          # creates a new, reviewable commit
git push                                              # opens/updates the PR
# PR reviewed like any other change (or expedited via the emergency path)
# on merge: Argo CD's application-controller detects the Git change,
# computes the diff, and reconciles automatically
argocd app get checkout-api                           # confirm sync completed successfully

No separate "rollback" button or pipeline stage exists; the revert PR IS the rollback, reviewed and applied through the identical mechanism every other change uses.

Trade-offs and pitfalls

• Common mistake: treating a GitOps revert as automatically risk-free just because "it's just undoing something." As with a Terraform rollback, a revert's ACTUAL computed effect depends on current state, which may have moved since the original bad commit; skipping the normal plan/diff review specifically because "it's a rollback" is exactly backwards, a rollback deserves the SAME scrutiny as any change, not less.
• Imperative CI/CD's separate rollback mechanism is sometimes FASTER in the moment (a single button/command) but usually LESS reviewed, a real trade-off worth naming explicitly rather than assuming GitOps's model is strictly superior in every dimension; GitOps's rollback is slower by exactly the amount of review it goes through, which is the SAME safety property normal changes get, not an accident.
• A revert PR can itself partially fail to apply, exactly like any other forward change can, treating "the revert PR merged" as equivalent to "the rollback completed" skips confirming the reconciliation itself actually succeeded, a real, easy-to-skip verification step.
• The emergency-change path is available for a GENUINELY urgent rollback, but even that path preserves SOME review (a lightweight, in-the-moment approval) and a mandatory retroactive capture; it is a faster lane through the same fundamental model, not an exception that abandons GitOps's core review-everything property entirely.

Direct answer
Without a service mesh, a canary on native Kubernetes primitives means running a SEPARATE, smaller ReplicaSet for the canary version, sharing the same Service (and therefore the same label selector) as the stable ReplicaSet, and controlling the traffic split purely by the RATIO of canary-to-stable pod counts, since without a mesh there's no fine-grained percentage-based routing, only "however many pods exist, roughly that share of traffic."

Structured elaboration

1. Two ReplicaSets, one Service: the stable Deployment/ReplicaSet and a separate canary ReplicaSet both carry a label the shared Service selects on (e.g. app: checkout-api), so Kubernetes' built-in load-balancing (round-robin across matching endpoints) sends traffic to both, with the SPLIT determined by relative pod count, not an explicit percentage.
2. Shifting traffic gradually: scale the canary ReplicaSet up (and optionally scale stable down proportionally) in steps, for example canary at 1 of 100 total pods (~1%), then 5 of 100 (~5%), then 25, then fully replacing stable. This is coarser-grained than a mesh's weighted routing (you're limited by pod-count granularity, especially at low replica counts) but requires no additional infrastructure.
3. Evaluating metrics: since both versions share one Service, you need the canary pods separately LABELED and QUERYABLE (a version: canary label alongside the shared app label) so your metrics system can filter and compare canary-specific metrics against stable, even though both are receiving traffic through the same Service.
4. Automating promotion/rollback: a script or controller that watches the canary-specific metrics, and on a pass, scales the canary ReplicaSet up (and stable down) to the next step; on a fail, scales canary back to zero and stable back to full, reverting via the same ReplicaSet-scaling mechanism.

Worked example
At 100 total desired pods: canary starts at 1 replica (stable at 99), giving roughly 1% of traffic via Kubernetes' round-robin balancing across matching endpoints. After a clean observation window, canary scales to 5 (stable to 95), then 25/75, then finally canary fully replaces stable (canary scales to 100, stable to 0, and the canary Deployment is effectively promoted to become the new "stable").

Trade-offs and pitfalls
The coarse granularity is the real limitation: at low total replica counts, you can't achieve a genuinely fine percentage (with 10 total pods, the smallest non-zero canary slice is 10%, not 1%), and Kubernetes' round-robin isn't a precise, deterministic percentage split the way a mesh's weighted routing is, it's a rough approximation based on endpoint count. This approach is a reasonable, infrastructure-light starting point for teams without a service mesh, but the imprecision and lack of built-in session affinity or fine-grained routing rules are exactly what a mesh (or a load-balancer with native weighted routing, like an ALB) is built to solve properly.

Direct answer
Blast radius is how much of your system, and how many users, are exposed to a bad deployment before you can stop it. Reducing it means never letting a single change reach 100% of traffic or 100% of your infrastructure in one step: you deploy to a small slice first, isolate that slice from the rest, and give yourself controls that can cut it off fast.

Structured elaboration
Techniques, roughly cheapest-to-hardest:

• Small-batch / percentage rollouts: canary a change to 1-5% of traffic or instances before going wider, so a bug affects a small fraction of users instead of everyone.
• Resource isolation: run the new version in separate compute (a distinct pod set, node pool, or availability zone) so a resource-exhaustion bug in the new version can't starve the old version's capacity too.
• Traffic controls: circuit breakers that stop routing to a demonstrably unhealthy instance, and rate limiters that cap how much load any single new component can absorb before it's proven stable.
• Region/cell isolation: for a global service, containing a rollout to one region or one "cell" of a sharded architecture means a bad release can't take down every region at once.
• Feature flags: decoupling "deployed" from "exposed" means you can turn a specific feature off instantly without a full redeploy, which is a much smaller and faster blast-radius-reduction lever than rolling back code.

For a monolith specifically, blast radius reduction is harder because there's no natural unit smaller than "the whole app": the levers become instance-level canarying (a subset of instances behind the load balancer run the new build) and feature flags around risky code paths, since you can't isolate one internal module's resource usage the way you can with a separate microservice.

Worked example
A change to a recommendation algorithm rolled out to 2% of traffic in one region first. A latency regression showed up only under that region's specific traffic mix (a caching quirk tied to timezone-driven request patterns); because it was contained to 2% of one region, the fix-and-redeploy cycle affected a small, recoverable slice of users instead of the whole global user base.

Trade-offs and pitfalls
More blast-radius controls mean more operational complexity and slower time-to-full-rollout, so teams calibrate the aggressiveness of containment to the risk of the change: a config tweak might skip straight to 100%, while a payment-logic change might go through five separate stages. The pitfall is applying the same heavy process to every change regardless of risk, which erodes the very safety discipline it's meant to protect by making people route around it under deadline pressure.

Direct answer
For stateful services like session stores or databases, blue-green, rolling, and canary all face the same underlying tension: the data layer usually can't be duplicated or partially exposed the same way stateless compute can, so whichever strategy you pick has to reckon with version skew between old and new code reading and writing the SAME underlying data, not just switching which code is running.

Structured elaboration

• Blue-green: typically shares ONE data store between blue and green (duplicating a stateful backend is expensive and risks its own consistency problems), so the "instant switch" property only really applies to the stateless application layer; any schema or data-format change still needs its own backward-compatible migration discipline, since both environments may briefly need to work against the same data during validation. Consistency impact: low, IF the shared store's schema is kept compatible throughout; latency impact: minimal, since there's no gradual traffic-mixing period to reason about.
• Rolling: explicitly runs old and new code concurrently against the shared data store for the DURATION of the rollout (potentially minutes), which is the longest sustained version-skew window of the three strategies; this makes rolling the strategy MOST dependent on strict backward/forward compatibility being correct, since a meaningful fraction of the rollout duration has both versions live simultaneously.
• Canary: version skew exists too, but confined to a smaller fraction of traffic for the duration of the canary window, so the BLAST RADIUS of a compatibility bug is smaller even though the skew risk itself is conceptually the same as rolling's.
• Version-skew risk, common to all three: a write from new-version code needs to be correctly readable by old-version code (and vice versa) for as long as both are live; this is the same backward-compatible-migration discipline used for schema changes generally, just now unavoidable rather than optional, because SOME period of mixed-version operation is inherent to rolling and canary, and even blue-green's shared-store model can't fully escape it during validation.
• Read/write consistency models: for a session store or cache, eventual consistency between old and new code's writes is often tolerable (a slightly stale session read is usually a minor UX issue, not correctness-critical); for a database backing genuinely critical state (financial records, inventory counts), the SAME version-skew window demands much stricter guarantees, which is why the practical mitigation (feature flags decoupling the data-format change from the code deploy, versioned APIs, dual-read/dual-write patterns) needs to scale with how consequential a consistency violation actually is for that specific data.
• Client compatibility: clients (including OTHER services calling this one) that were built against the old version's data contract need to keep working during the transition; a versioned API contract, rather than an implicit shared understanding of the data shape, is what actually protects them.

Worked example
A session store migrating its session-serialization format: under ROLLING deployment, old and new application instances both read and write sessions concurrently for the full rollout duration, so the serialization format change must be backward AND forward compatible for that entire window (old code must tolerate a session written by new code, and new code must tolerate one written by old code). Under CANARY, the same compatibility requirement applies but only affects the canary's small traffic slice, so a compatibility bug's blast radius is much smaller even though the underlying risk is identical. Under BLUE-GREEN sharing one session store, the risk window shrinks to the validation period before cutover plus any drain period after, generally the shortest exposure of the three, but not zero.

Trade-offs and pitfalls
The common mistake is assuming blue-green "solves" the stateful-service problem the way it solves the stateless one, when in practice the shared data layer still carries real version-skew risk, just over a shorter window; the strategies differ in HOW LONG and HOW BROADLY that risk window is open, not in whether it exists at all. Mitigation patterns (versioned APIs, dual-read/dual-write, feature-flag-gated format changes) apply across all three strategies and are what actually manage the risk, the choice of deployment strategy mainly changes the window's duration and blast radius, not whether the mitigation is needed.