Deployment Risk Management & Rollback Strategy Questions

Discuss strategies for managing deployment risk: canary deployments (detect issues in subset), feature flags (quick disable without rollback), smoke testing post-deployment. Understand rollback procedures: full rollback (restore previous version), partial rollback (revert specific services). Know how to handle complications like database schema changes that can't simply rollback.

HardSystem Design

43 practiced

Design CI/CD pipeline patterns to support zero-downtime database schema migrations when multiple service versions may be live concurrently. Include migration phases, feature flags, coordination across repositories, canarying of migration effects, and pipeline gates that block unsafe promotions.

Sample Answer

**Problem framing & goals**Support zero-downtime schema changes while multiple service versions may run: safe forward/backward compatibility, automated coordination, observable canaries, and pipeline gates to stop unsafe promotions.

**Migration phases (pipeline stages)**- Pre-migration (CI): run DB migration linter, backward-compatibility checks, integration tests against a cloned schema.- Deploy-safe migration (deploy phase 1): apply additive migrations (new tables/columns, nullable columns, triggers off). Record version and idempotency metadata.- Dual-write phase: services opt into writing new fields while continuing to read old ones. Controlled via feature flags.- Backfill/transform: run background jobs to populate new columns; monitor consistency metrics.- Read-switch & cleanup: switch reads to new fields, remove old fields after sufficient canary and rollback window.

**Feature flags & versioning**- Use runtime feature flags (LaunchDarkly/Flipper) per service to toggle writes/reads.- API contract versioning: tag images with schema-compatibility metadata. Migrations tagged with schema-version.

**Cross-repo coordination**- Central migration registry repo with migration manifests (id, preconditions, rollback script, compatible service versions).- Each service pipeline fetches manifest, validates compatibility, and blocks deployment unless migration state permits.

**Canarying and safety gates**- Canary deployments of services + limited-rollout migration target subset of DB shards or tenant subset.- Observability: DB telemetry (latency, error rates, data divergence), business KPIs, and custom consistency checks.- Pipeline gates: automated checks that must pass before promotion — linter, compatibility tests, canary health, backfill lag thresholds. Human approval required for destructive ops.

**Rollback & runbook**- Every migration includes automatic rollback path and manual runbook. Use feature flags to disable writes instantly and revert reads.

This pattern balances automation with safe human oversight, enabling zero-downtime migrations across multiple live service versions.

MediumTechnical

47 practiced

Write a Kubernetes YAML snippet (or Helm values) that defines a canary traffic split using Istio VirtualService (or equivalent) to route 10% of traffic to a canary subset labeled "canary: true" and 90% to the stable subset. Include the Service/Deployment label selectors and VirtualService traffic weights.

Sample Answer

**Approach**- Use a Service selecting pods by app label; create two Deployment manifests that set a pod label (canary: "true"/"false").- Define an Istio DestinationRule with subsets matching those labels.- Define an Istio VirtualService that splits traffic 90/10 between stable and canary subsets.

**YAML example**

yaml

# Service
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    name: http

---
# Stable Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-stable
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      canary: "false"
  template:
    metadata:
      labels:
        app: my-app
        canary: "false"
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:stable
        ports:
        - containerPort: 8080

---
# Canary Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      canary: "true"
  template:
    metadata:
      labels:
        app: my-app
        canary: "true"
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:canary
        ports:
        - containerPort: 8080

---
# Istio DestinationRule with subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app-destination
spec:
  host: my-app
  subsets:
  - name: stable
    labels:
      canary: "false"
  - name: canary
    labels:
      canary: "true"

---
# Istio VirtualService with 90/10 split
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app-virtualservice
spec:
  hosts:
  - my-app
  http:
  - route:
    - destination:
        host: my-app
        subset: stable
      weight: 90
    - destination:
        host: my-app
        subset: canary
      weight: 10

**Key notes / best practices**- Ensure DestinationRule subsets match pod labels exactly.- Start canary small (10% or less), monitor metrics (latency, errors, logs).- For Helm values: expose image tags, replica counts, and canary weight as values to control rollout dynamically.- Consider traffic mirroring or header-based routing for debugging, and automatic rollback if errors spike.

EasyTechnical

42 practiced

What key metrics and signals would you monitor during a canary deployment to decide whether to promote or rollback? Include system-level, application-level, and business-level metrics and explain why each matters for making the decision.

Sample Answer

**Approach (brief)** As a DevOps engineer I monitor a blend of system, application, and business signals during a canary and apply SLO-based thresholds + statistical checks (confidence interval / sample size) before promoting or rolling back.

**System-level metrics**- CPU / Memory / Disk I/O / Network throughput — detect resource saturation or regressions that cause instability.- Pod/container restart rate & OOMs — immediate red flags for crashes or memory leaks.- Host-level errors (kernel, disk) — indicate infrastructure issues unrelated to code.

Why: infrastructure problems can silently affect the canary even if app metrics look OK.

**Application-level metrics**- Error rate (5xx, application exceptions) — primary safety signal; compare against baseline and SLO (e.g., >2x baseline or >SLO).- Latency (p50/p95/p99) — increases often precede errors; use percentiles to catch tail latency.- Throughput / requests per second — ensures comparable traffic to baseline; low sample size may invalidate results.- Success rate for critical flows (health checks, DB calls, downstream latency) and saturation of resource pools (DB connections).- Logs and traces (error anomalies, stack traces) — root cause insight.

Why: these directly reflect user-facing reliability and performance.

**Business-level metrics**- Conversion rate, checkout success, active sessions, revenue per minute — detect user impact not visible via infra/app metrics.- User-facing latency (page load times) and customer complaints / support tickets.

Why: even small technical regressions can harm revenue or UX; business metrics validate real impact.

**Decision rules**- Promote if canary meets SLOs, no statistically significant regression vs baseline, and business metrics stable.- Roll back if error rate or p95/p99 latency exceeds thresholds, resource exhaustion, or negative business signals.- Include automatic short-circuit rollback for hard failures (crash loops, data loss) and gradual ramp with hold windows and manual review when borderline.

**Notes**- Ensure adequate traffic and run-time for statistical confidence (e.g., minimum requests or minutes).- Use anomaly detection and automated alerts to reduce detection time.

EasyTechnical

48 practiced

What is a backward-compatible database migration? Provide examples of safe (backward-compatible) and unsafe (non-backward-compatible) schema changes, and explain why backward compatibility is critical for safe rollbacks and phased deployments.

MediumTechnical

46 practiced

How would you design automated tests and manual rehearsals in staging to validate rollback procedures? Describe which elements to automate (e.g., scripts, data backups, restore tests) and which to rehearse manually (e.g., cross-team communication), how to measure success, and how frequently to run these rehearsals.

Unlock Full Question Bank

Get access to hundreds of Deployment Risk Management & Rollback Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.