Covers end to end practices, automation, and architectural choices for delivering software safely and frequently. Candidates should understand and be able to compare deployment and upgrade approaches such as blue green deployment, canary releases, rolling updates, recreate deployments, shadow traffic and shadow deployments, and database migration techniques that avoid downtime. This topic includes progressive delivery and feature management practices such as feature flagging, staged rollouts by user cohort or region, staged traffic ramp up, and progressive delivery platforms. Candidates should be able to explain safety controls and verification gates including health checks, automated validation gates, smoke testing and staging verification, automated rollback criteria, and emergency rollback procedures. They should understand zero downtime patterns, rollback complexity and mechanisms, capacity and resource requirements, latency and consistency trade offs, and techniques to reduce blast radius and deployment risk. The topic also covers release engineering and operational practices such as release orchestration across environments, deployment automation and pipelines, continuous integration and continuous delivery practices, approvals and release management processes, incident response and communication during releases, chaos testing to validate resilience, and observability and monitoring to detect regressions and measure release health. Candidates should be able to describe metrics to measure deployment velocity and reliability such as deployment frequency, mean time to recovery, and change failure rate, and explain how to design frameworks, automation, and operational processes to enable frequent safe deployments at scale.
MediumTechnical
135 practiced
Create a release checklist and approvals workflow for a regulated industry (e.g., finance or healthcare) that requires audit trails, sign-offs, and emergency rollback capability. Include automation to reduce human error while meeting compliance.
Sample Answer
Overview: I propose a release checklist + approvals workflow that enforces separation of duties, creates immutable audit trails, automates safety gates, and provides fast emergency rollback. It uses pipeline-driven promotion, policy-as-code, and documented human sign-offs stored in the change system.Release checklist (pre-release):- Code merged via PR with required approvers and CI passing (unit, integration, static code analysis, SCA).- Build produces signed immutable artifact (Artifactory/Nexus + GPG checksum).- Automated infrastructure drift check (Terraform plan + policy checks via OPA).- Automated security & compliance scans (SAST/DAST, infra CIS benchmarks).- Backups and DB migration dry-run validated; runbook updated.Approvals workflow:- Pipeline creates a Change Request in Jira/ServiceNow with artifact metadata, diffs, risk score.- Mandatory approvers: Dev Lead, QA, Security, Compliance, and Release Manager. Approvals recorded with SSO+MFA and timestamp; stored as immutable entries in audit log (WORM storage / Splunk with append-only retention).- Low-risk changes use automated gating; high-risk require explicit CAB sign-off. Emergency-only path: emergency CAB (on-call + compliance rep) can approve via expedited flow; approval recorded and flagged.Automated deployment & safety:- Promote via CD tool (ArgoCD/Spinnaker) using environment manifests (Helm/Kustomize).- Canary deployment with real-time health checks and SLO-based automated rollback thresholds.- Feature flags for DB/schema toggles and progressive exposure.- Automatic rollback: if health or error budget breached, orchestrated rollback executed (K8s rollout undo, revert helm release) plus DB rollback procedures if reversible; if not reversible, circuit off feature flags and alert on-call.Audit & traceability:- Artifact provenance, signed manifests, pipeline logs, approval metadata, and deployment events are centrally logged with immutable retention and indexed for audit.- Use policy-as-code (OPA/Gatekeeper) to prevent unauthorized configs; secrets in Vault; access via RBAC and reviewed quarterly.Post-release:- Automated smoke tests, synthetic and real-user monitoring dashboards, post-release sign-off recorded when metrics stable.- Blameless postmortem for incidents; lessons feed back into automated checks.This balances automation (reducing human error) with documented human sign-offs and fast, safe emergency rollback while producing a complete auditable trail for regulators.
EasyTechnical
80 practiced
Describe a canary release. What are the essential components of a canary pipeline (traffic split, metrics, analysis) and how do you decide the initial canary size, ramping increments, and stop/rollback criteria?
Sample Answer
A canary release is a progressive-delivery pattern where a new version is rolled out to a small subset of users or traffic first, monitored for problems, and then gradually ramped to full production if metrics remain healthy. It reduces blast radius and lets you validate behaviour in production.Essential components of a canary pipeline:- Traffic split: ability to route a precise fraction of traffic (1–50%) to the canary using feature flags or networking (Ingress, Service Mesh — Istio/Linkerd, Kubernetes + Argo/Flagger).- Metrics: baseline and canary metrics including platform (error rate, latency p95/p99, CPU/memory), business (conversion, checkout rate), and logs/traces/synthetics.- Analysis: automated comparison (statistical tests, confidence intervals, anomaly detection, or canary scoring tools like Kayenta/Flagger) plus dashboards and alerts.Deciding initial canary size and ramps:- Start small for risky changes: 0.5–2% for backend/core infra, 2–5% for safer UI tweaks. Consider traffic diversity: ensure sample reaches affected code paths.- Ramp strategy: linear or exponential increments (e.g., 1% → 5% → 20% → 50% → 100%) with dwell time per step (minutes to hours) equal to one or more full user behavior cycles and enough requests for statistical power.- Use SLOs and failure budgets to set max ramp and pace: slower if close to budget.Stop/rollback criteria:- Predefined, measurable thresholds: error rate spike beyond X× baseline (e.g., >2× or absolute >0.5%), latency degradation (p95/p99 increase >20%), CPU/memory saturation, or negative business metric delta beyond acceptable bounds.- Statistical significance: require confidence (e.g., p < 0.05) before declaring canary healthy; if insufficient data, hold instead of advancing.- Automated actions: halt ramp, shift traffic back, trigger alerts, create incident ticket and capture diagnostics (logs, traces, heap dumps).- Safety nets: circuit breakers, kill-switch feature flags, and manual approval gates for risky steps.This approach balances speed with safety: small, measurable exposure, automated analysis, and clear rollback rules tied to SLOs.
HardSystem Design
99 practiced
You are responsible for scaling a canary rollout system so it can run concurrently for 200 services without resource contention. Explain how you'd design rate-limiting, shared analysis infrastructure, caching of metrics, and isolation to avoid noisy neighbor effects.
Sample Answer
Requirements & constraints:- Run concurrent canaries for 200 services with minimal latency and no cross-tenant interference.- Preserve timely analysis (near real-time), bounded cost and predictable tail latency.- Support metric-heavy workloads (high-cardinality time-series queries) and statistical analysis jobs.High-level approach:- Enforce multi-level rate limits, build a shared but sharded analysis service, aggressively cache/pre-aggregate metrics, and isolate resource usage via Kubernetes + quota controls and fair scheduling.Design details1) Rate limiting (multi-level, hierarchical)- Per-service soft quota: e.g., token-bucket of N analysis-queries/min per service (configurable by SLA).- Global pool + priority: reserve a small global quota for high-priority/rollback actions, remaining on shared pool.- Enforcement points: - Client-side (canary orchestrator) to prevent bursts. - Edge proxy (Envoy) or centralized limiter using Redis (Leaky/Token bucket) for cross-process correctness.- Backpressure & graceful degradation: when quota exhausted, return "deferred" result and schedule sampling later; expose metrics to tune rates.2) Shared analysis infrastructure (scalable, sharded)- Stateless analysis API frontends behind autoscaling LB; worker pool for heavy computations.- Shard analysis by service or by metric namespace to localize hot tenants. Use consistent hashing to keep same service routed to same worker subset.- Worker pool autoscale based on queue length and CPU.- Use job priority queues (Rabbit/Kafka) and rate-limited consumers to avoid overload.- Lightweight preflight checks: cheap anomaly heuristics to reject obviously noisy/invalid canaries before full analysis.3) Metrics caching & pre-aggregation- Use time-series DB (Prometheus remote-write -> Cortex/Thanos/M3DB) + query-frontend with read-through cache (Redis or CDN) for query results.- Pre-aggregate rollups: 10s raw for 1h, 1m rollups for 7d, 5m rollups longer. Serve rollups for canary evaluation where acceptable.- Query caching keys: (service, metric, window, aggregation, version). TTL ≈ analysis frequency; invalidate on new ingestion if needed.- Materialize common feature sets (latency quantiles, error rates) into derived storage so analysis jobs read compact records instead of raw TS.4) Isolation & noisy-neighbor controls- Kubernetes namespaces per team + ResourceQuota + LimitRanges. Enforce CPU/memory requests+limits for analysis pods and canary orchestrators.- cgroups/CPU shares and node taints: dedicate a pool of nodes for canary analysis (isolate heavyweight work) and a low-latency pool for orchestrator control plane.- Pod eviction and quality-of-service tiers: best-effort vs guaranteed.- Fair-share scheduling at job queue level (weight per tenant) to ensure no single service consumes all workers.- Circuit-breaker per-tenant: detect repeated heavy jobs and throttle or convert to sampled runs.Operational considerations & trade-offs- Observability: track per-service quota usage, latency percentiles, cache hit rates, analysis queue depth. Alert on sustained throttling.- Consistency vs freshness: caching/rollups trade freshness for throughput; allow emergency “full-fidelity” runs that bypass rollups at higher cost.- Cost vs isolation: dedicated nodes reduce noisy neighbors but increase cost; sharding + fair-share often balances both.Example knobs to tune (for 200 services)- Default per-service concurrent analyses = 2; burst up to 5 if global capacity allows.- Worker pool: scale to handle peak of 400 analysis tasks; shard into 20 worker groups (10 services/group average).- Cache TTL = 30s for 1m rollups; hit rate target >80% to keep load manageable.This design yields predictable resource usage, low contention through hierarchical rate-limiting, efficient reads via caching/rollups, and strong isolation via scheduling, quotas and sharding—while keeping options to bypass caches for urgent full-resolution analysis.
MediumTechnical
90 practiced
As a DevOps engineer, how would you measure and improve change failure rate for your platform? Propose a set of experiments, process changes, and tooling adjustments to reduce failures over a quarter.
Sample Answer
Situation: Our platform’s Change Failure Rate (CFR) was ~12% last quarter—too many rollbacks and urgent fixes that harmed reliability and developer velocity.Task: I needed to measure CFR precisely, run experiments and make process/tool changes to cut CFR by at least half within one quarter while preserving deployment frequency.Action:- Measurement & baseline (weeks 1–2): Defined CFR = failed releases (deploys causing incident/rollback/severity ≥ P2) ÷ total deploys. Instrumented CI/CD and incident DB to tag each change with deploy ID, owner, and outcome; added a CFR dashboard and correlated MTTR, lead time, and deployment size.- Quick experiments (weeks 3–6): - Canary vs full deploy A/B: roll out 5% traffic canary for high-risk services using Kubernetes + Argo Rollouts; measure failure detection rate and rollback time. - Feature flags on new behavior (LaunchDarkly/Unleash): enable gradual ramp and instant kill-switch. - Pipeline gates: add smoke tests and contract tests that run against ephemeral environments before promotion.- Process changes (weeks 5–10): - Require readme/runbook updates and readiness checklist for on-call sign-off for risky changes. - Enforce small, single-purpose PRs and trunk-based development to reduce change size. - Blameless postmortems for every failed change with a corrective action owner and timeline. - Introduced "pre-deploy chaos" on canaries for critical services to surface hidden assumptions.- Tooling adjustments (weeks 3–12): - Improve observability: distributed tracing, error-rate SLOs, and automated alert thresholds tied to deploy IDs. - Automated rollback policies: rollback when error rate > X% or latency degrades Y% for T minutes. - CI optimizations: parallelized integration tests, faster feedback loops, and staged approvals in pipeline. - Templates and automation for runbooks, incident creation and tagging.- Evaluation & iterate (weeks 11–12): Compare CFR, MTTR, and deployment frequency; run retros and refine thresholds.Result: Expected outcome: CFR reduced from ~12% to <6%, MTTR down ~30–50%, and deployment frequency maintained or improved. The combination of measurable baseline, progressive delivery experiments, tighter pipeline gates, better observability, and enforced lightweight processes creates sustainable improvement and a feedback loop to keep lowering CFR.
MediumTechnical
90 practiced
You need to perform a zero-downtime schema migration adding a non-nullable column with default on a large table (hundreds of millions of rows). Outline a safe migration approach with steps, required application changes, and verification checks.
Sample Answer
Clarify constraints: non-nullable column with a default, zero downtime, hundreds of millions of rows, no blocking table rewrite or long locks.High-level approach (safe, phased):1. Assess DB behavior and safety tools- Confirm how your RDBMS handles ADD COLUMN with DEFAULT (some versions rewrite; Postgres 11+ optimizes; MySQL may block).- Prepare backups, ensure replicas can handle load, and have rollback plan.2. Phase A — Schema additive, non-blocking- Add the column as nullable and WITHOUT a default so the DDL is fast/non-blocking:
sql
ALTER TABLE users ADD COLUMN new_flag boolean;
3. Phase B — App changes (dual-read/write)- Deploy app version that: - Writes to new_flag on all mutating paths (dual-write). - Reads prefer new_flag but fall back: COALESCE(new_flag, <default>) so behavior unchanged.- Use feature flag to gate changes.4. Phase C — Backfill in controlled batches- Backfill existing rows in small transactions to avoid long locks and replication lag. Example pattern:
sql
-- pseudo: iterate ranges or use primary-key CTE chunks
WITH chunk AS (
SELECT id FROM users WHERE new_flag IS NULL ORDER BY id LIMIT 10000
)
UPDATE users u SET new_flag = true FROM chunk c WHERE u.id = c.id;
- Throttle concurrency, monitor replication lag, CPU, I/O.5. Phase D — Verification- Ensure zero NULLs: SELECT count(*) FROM users WHERE new_flag IS NULL;- Spot-check consistency: sample rows, checksums between old behavior and new_flag.- Monitor app logs for errors, increased latency, DB metrics, and replica lag.6. Phase E — Finalize schema- Once backfill finishes and app stable:
sql
ALTER TABLE users ALTER COLUMN new_flag SET DEFAULT true;
ALTER TABLE users ALTER COLUMN new_flag SET NOT NULL;
(If your DB rewrites on these, do them during low impact window or use DB-specific online ALTER.)7. Cleanup- Remove fallback read logic and feature flag; remove any temporary scripts.Safety gates and operational controls- Circuit-breaker in backfill script (stop on high lag/error).- Throttling and pause/resume.- Pre-migration load test on staging-sized dataset.- Continuous monitoring: replication lag, long transactions, index usage, error rates.- Rollback plan: stop backfill, revert app writers, remove column writes; if catastrophic, restore from backup.Trade-offs and alternatives- For MySQL, consider gh-ost or pt-online-schema-change if ALTER blocks.- If immediate non-null default is required and DB supports fast default, you can set default earlier — but verify version behavior.This phased, dual-write + backfill pattern ensures application correctness, no downtime, and safe progression to a strict NOT NULL column.
Unlock Full Question Bank
Get access to hundreds of Deployment and Release Strategies interview questions and detailed answers.
