Deployment Risk Management & Rollback Strategy Questions

Discuss strategies for managing deployment risk: canary deployments (detect issues in subset), feature flags (quick disable without rollback), smoke testing post-deployment. Understand rollback procedures: full rollback (restore previous version), partial rollback (revert specific services). Know how to handle complications like database schema changes that can't simply rollback.

EasyTechnical

62 practiced

Explain what an error budget is and how it should influence deployment risk and release gating. Provide a concrete example: propose an SLO, translate it into an error budget, and describe how you'd change deployment behavior when the error budget is nearly exhausted.

EasyTechnical

58 practiced

List three common failure modes encountered by post-deployment smoke tests (e.g., connectivity errors, configuration drift, auth regressions). For each, explain how you'd detect it with monitoring and how to automatically respond to minimize customer impact.

Sample Answer

1) Connectivity errors (e.g., downstream service or DB unreachable)Detection:- Synthetic smoke test that performs a simple read/write or health-check across the critical path.- Monitor: high error rate or increased latency on the synthetic check, connection timeout metrics, TCP connection failures, and downstream 5xx rates.Auto-response:- Circuit-breaker: automatically route traffic to healthy replicas or fallback service (read-only mode).- Trigger automated rollback or deploy a previously stable release if failures coincide with a recent deploy.- Open a paging alert for on-call only when synthetic failure persists beyond a short grace period (e.g., 1–2 minutes) to avoid noisy alerts.

2) Configuration drift (wrong feature flags, stale secrets, misconfigured routings)Detection:- Monitor: divergence between expected config manifest (git) and live config (periodic config-compare), failed template rendering, auth/permission errors in smoke tests.- Alert when checksum mismatch or when smoke test exposes different behavior than baseline.Auto-response:- Automated reconcilier: reapply desired config from source-of-truth and restart affected pods/services.- If reapply fails or causes instability, revert to previous known-good config and notify owners.

3) Authentication/authorization regressions (token failures, revoked keys, scope changes)Detection:- Smoke tests exercising login flows and token refresh paths; monitor OAuth error rates, 401/403 spikes, and token-expiry logs.- Correlate with secrets manager rotation events or recent IAM policy changes.Auto-response:- Automatically retry with refreshed credentials from secret store; rotate to secondary key if primary fails.- Temporarily put impacted endpoints into degraded mode (serve cached/limited content) and alert engineers.- If rollback is safe, revert the change that caused the auth regression.

Across all modes:- Rate-limit automated actions and require human confirmation for destructive steps.- Record all automated steps in incident timeline and expose metrics for SLO error-budget accounting.

EasyTechnical

42 practiced

Describe feature flags (feature toggles): types (release flags, ops flags, experiment flags), lifecycle (creation, targeting, cleanup), common risks (config drift, stale flags), and operational practices SREs should follow when flags are used extensively in production.

HardBehavioral

48 practiced

Describe a time you led a cross-functional rollback or emergency remediation under high pressure. If you don't have a direct example, outline how you would assemble the team, establish communication, make go/no-go decisions, execute the rollback, and run the post-incident review to ensure learning and follow-up actions.

Sample Answer

Situation: Two months ago during a weekday peak we deployed a release that introduced a schema change and a sidecar that caused pod restarts and cascading 5xx errors across three services. Customer-facing latency spiked 4x and error budget burn was rapid; pages were firing.

Task: As on-call SRE and incident commander, I needed to stop user impact quickly, coordinate a cross-functional rollback, and restore safe service levels while preserving data integrity.

Action:- Assembled a rapid response team: one engineering lead from each affected service, a DB engineer, release manager, QA, and two SREs (one as Incident Commander (IC), one as Runbook executor). I assigned roles immediately (IC, communications lead, rollback executor, telemetry lead).- Established communications: created an incident Zoom + persistent Slack channel, posted concise incident header (impact, hypothesis, ETA cadence) and a dedicated PagerDuty status note. Began 5‑minute update cadence.- Triage & decision: telemetry lead confirmed error spike aligned with deploy timestamp and DB migration warnings; rollback risk assessed (data-backwards compatibility checked with DB owner). Using the predefined rollout playbook and error-budget threshold, I declared a go for rollback within 15 minutes.- Execute rollback: SREs used the deployment system to rollback the Kubernetes release to the previous stable image and disabled the feature flag. We monitored CPU, pod restarts, 5xx rate, and user-facing latency; DB changes were non-destructive so no schema revert required. Rollback completed in 12 minutes; error rate returned to baseline within two propagation cycles.- Communication and stakeholder updates: sent an incident summary to execs and CS with timeline and mitigation steps; kept 5‑minute updates until stability.- Post-incident review: within 48 hours I led a blameless postmortem with dev, DB, QA, and release teams. Root causes: insufficient canary coverage, missing migration safety checks, and an untested sidecar interaction. Actions: require automated canary traffic for DB migrations, add migration compatibility checks to CI, add pre-deploy chaos tests for sidecars, improve runbook rollback steps, and schedule a follow-up QA sign-off gate. I owned tracking the action items in our backlog and closed 6/7 items within two sprints.

Result: User latency and 5xxs returned to SLOs within 30 minutes of incident start. Error budget burn was limited, customer impact minimized, and the new CI gates prevented recurrence. The process improvements reduced similar deploy-related incidents by 60% over the next quarter.

This experience reinforced the value of clear roles, rapid evidence-based go/no-go decisions, automated canaries, and actionable postmortems to convert pressure into durable reliability improvements.

MediumTechnical

57 practiced

You're an SRE who finds that the release process lacks formal rollback policies and engineers are reluctant to pause releases. How would you influence product and engineering teams to adopt SLOs, rollback runbooks, and automated gates while preserving reasonable velocity?

Unlock Full Question Bank

Get access to hundreds of Deployment Risk Management & Rollback Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.