Covers end to end practices, automation, and architectural choices for delivering software safely and frequently. Candidates should understand and be able to compare deployment and upgrade approaches such as blue green deployment, canary releases, rolling updates, recreate deployments, shadow traffic and shadow deployments, and database migration techniques that avoid downtime. This topic includes progressive delivery and feature management practices such as feature flagging, staged rollouts by user cohort or region, staged traffic ramp up, and progressive delivery platforms. Candidates should be able to explain safety controls and verification gates including health checks, automated validation gates, smoke testing and staging verification, automated rollback criteria, and emergency rollback procedures. They should understand zero downtime patterns, rollback complexity and mechanisms, capacity and resource requirements, latency and consistency trade offs, and techniques to reduce blast radius and deployment risk. The topic also covers release engineering and operational practices such as release orchestration across environments, deployment automation and pipelines, continuous integration and continuous delivery practices, approvals and release management processes, incident response and communication during releases, chaos testing to validate resilience, and observability and monitoring to detect regressions and measure release health. Candidates should be able to describe metrics to measure deployment velocity and reliability such as deployment frequency, mean time to recovery, and change failure rate, and explain how to design frameworks, automation, and operational processes to enable frequent safe deployments at scale.
HardSystem Design
71 practiced
Design a deployment architecture for a global application with 100M users requiring <100ms latency SLAs. The architecture must support zero-downtime deployments across regions, progressive rollouts, and acceptable data consistency for user data. Describe traffic routing, data replication strategies, deployment patterns, and verification steps during rollouts.
Sample Answer
Requirements (clarify): global 100M users, <100ms user-perceived latency, zero-downtime cross-region deploys, progressive rollouts, "acceptable" (multi-second) consistency for user data.High-level architecture:- Global edge (CDN + Anycast DNS) → Global API Gateway / Traffic Orchestrator → Regional clusters (K8s/ECS) in 6+ Regions (active-active) → Regional caches (Redis/Memcached) + regional read/write DBs (primary-local, async replicas) → Centralized control plane for deployments and feature flags.Traffic routing:- Use Anycast + geo-DNS for low-latency routing to nearest region.- Health-aware global load balancer (e.g., GSLB) with latency and capacity probes; failover to next-best region if SLAs breached.- Session affinity only when needed; otherwise stateless tokens (JWT) or short-lived session caches replicated.Data replication / consistency:- Use per-user "home region" mapping for strong-ish writes: write-local primary in home region, async geo-replication for other regions (eventual consistency).- For low-latency reads, use regionally-cached reads with read-repair or version vectors.- For cross-region shared data requiring tighter consistency, use distributed consensus (e.g., CockroachDB/Spanner) for small critical tables; or a globally-ordered change stream (Kafka + log-compact topics) with last-write-wins for non-critical user prefs.Deployment patterns:- Blue/Green or Canary controlled by a central deployment controller (Argo Rollouts/Spinnaker).- Rollout strategy: Canary in single region → progressive canaries across regions (5%→25%→100%) with weighted routing.- Zero-downtime: use readiness/liveness probes, connection draining, and schema migrations that are backward-compatible (expand-contract pattern).Verification & safety gates:- Automated canary analysis: compare error rate, latency P95/P99, resource usage, business metrics (login success, purchase flow) vs baseline using metrics platform (Prometheus + Grafana + Kayenta/Argo).- Synthetic transactions from multiple regions (SLA probes) and real-user monitoring (RUM).- Automated rollback triggers: increased error rate, SLA breach, or degraded core business KPI.- Gradual schema migration verification: deploy new code that tolerates old schema, backfill, then remove old columns.Operational considerations / trade-offs:- Accept eventual consistency for most user data to meet latency. Use home-region writes to minimize conflicts.- Use distributed transactional DB only where correctness > latency.- Cost vs latency: more regions reduce latency but increase operational cost; optimize by colocating heavier read caches at edge.This design meets <100ms latencies via geo-routing and regional primaries, supports zero-downtime with blue/green + draining, and enables progressive, safe rollouts with automated verification and rollback.
MediumTechnical
67 practiced
Describe how you would orchestrate a coordinated release across multiple dependent services when one service introduces a breaking API change that requires consumers to be updated. Include API versioning strategy, deployment ordering, feature flags for toggling behavior, consumer adapters, and verification steps to avoid system-wide outages.
Sample Answer
Requirements & constraints:- Breaking change in Service A requires consumers B, C to update.- Zero/low downtime, reversible, testable, and coordinated across teams.Plan (high level):1. API versioning strategy- Introduce v2 of Service A (non-destructive): /v2/... or media-type versioning (Accept header).- Keep v1 available in read-only/maintenance mode for a deprecation window.- Document contract differences and migration guide.2. Consumer adapters and compatibility shim- Provide a lightweight adapter library for B/C that maps v2 payloads to v1 shapes (and vice versa), enabling incremental rollout.- Publish adapters as versioned client SDKs and containerized sidecar proxies if language diversity exists.3. Deployment ordering & orchestration- Deploy Service A v2 behind a feature flag (or route via API gateway) in parallel with v1.- Release adapters/SDKs to B/C and run them in “compatibility mode” using feature flags that select v2 endpoints.- Order: A(v2, toggled off) → Adapter release to consumers → Integration testing → Toggle A(v2) on for small percentage (canary) → Gradual traffic ramp.4. Feature flags & toggles- Use server-side flags (FF system) to control: - Whether A responds on v2 routes - Whether consumer uses adapter vs native parsing - Rollback switches per service- Flags allow quick rollback without redeploy.5. Verification & safety nets- Automated contract tests (Pact or contract testing) between A and each consumer.- End-to-end integration and canary tests in staging with synthetic traffic.- Observability: detailed metrics, tracing, error budgets, health checks, and alerting.- Circuit breakers & graceful degradation in consumers to prevent cascading failure.6. Rollback & deprecation- If issues, flip feature flag to route to v1; use adapters for interim fixes.- Communicate deprecation timeline and metrics to stakeholders; remove v1 after clients migrate.Why this works:- Versioning + adapters minimize immediate breakage.- Feature flags enable safe gradual rollout and fast rollback.- Contract tests and canaries catch integration regressions early, avoiding system-wide outages.
MediumTechnical
89 practiced
You are advising a high-traffic e-commerce customer preparing for Black Friday. They ask whether to use blue-green or canary deployments for a release that touches checkout and inventory services (heavy writes). As a Solutions Architect, evaluate both strategies, recommend an approach, and outline the operational runbook for the release, focusing on risk, rollback speed, and cost.
Sample Answer
Recommendation summary:- Use a phased canary release with feature flags and strict backward-compatible DB changes (expand-then-contract migration pattern). For checkout + inventory (heavy writes), canary minimizes blast radius and lets you validate write-path behavior against production data without dual-writes or DB divergence that blue‑green often incurs. Blue‑green is acceptable only if you can afford full duplicate capacity and guarantee synchronous state or safe cutover of write workloads — expensive and higher operational cost.Why canary is better here (risk, rollback speed, cost):- Risk: Canary limits affected users to a small percentage; you observe real writes and inventory reconciliation. Blue‑green risks data divergence if both environments accept writes or requires complex synchronization.- Rollback speed: Canary rollback is fast — cut traffic back to previous app instances or flip feature flag. Blue‑green rollback requires switching traffic too, but if DB migrations ran, rollback may be impossible without complex reverse-migration.- Cost: Canary uses existing capacity (incremental autoscaling) and small new instance pool; blue‑green requires full duplicate capacity during peak (high cost for Black Friday).Operational runbook (phased canary + safety controls)Pre-release (Days leading to BF)1. Architecture & data prep - Ensure DB migrations follow expand-then-contract pattern; no destructive schema changes. - Add feature flags to gate new checkout/inventory logic. - Ensure idempotency on critical write APIs and idempotency keys on checkout. - Add metrics/trace hooks: success rates, latency, error budgets, inventory delta reconciliation, payment gateway metrics, order duplication, reconciliation lag.2. Capacity & rollback readiness - Warm canary instances in same AZs; set autoscaling policies. - Prepare automation playbooks and runbooks for rollback and DB emergency scripts (e.g., toggle flag, freeze writes to new path). - Run load tests that simulate BF write rates on canary traffic mix.Canary release steps (day of release)1. Baseline (T-120m) - Verify monitoring, alerting, dashboards. - Snapshot DB or take relevant backups (point-in-time backups for critical tables). - Notify CS/ops/merchant teams; set on-call rotation.2. Canary 1 (T-60m) - Route 0.5–1% traffic to new version via load balancer / service mesh. - Gate new behaviors with feature flags; enable only for canary users. - Run synthetic purchase flows and manual QA on canary. - Observe for 15–30 minutes: - Latency, error rate, inventory mismatch rate, payment failures, duplicate orders, DB locks.3. Canary 2 (T+) - If stable, increase to 5% for 30–60 minutes. Continue monitoring. - Run reconciliation jobs comparing inventory aggregate changes between canary and baseline.4. Canary 3 (T+) - Increase to 20–25% for longer validation (1–2 hours). If stable, proceed to progressive ramp to 50% then 100% during low-risk window. - Maintain ability to cut back immediately via traffic control or feature flag.Rollback criteria (explicit)- Immediate rollback if: error rate > predefined threshold (e.g., >1% increase absolute), payment gateway failures, order duplication, inventory reconciliation drift beyond tolerance, increased DB contention/lock timeouts, or SLO breaches.- If criteria hit: toggle feature flag to old flow, drain canary instances, and revert traffic. If DB migration caused irreversible changes, invoke emergency compensation plan (manual reconciliation, freeze new orders if needed).Rollback steps (fast path)1. Toggle feature flag to revert to old code path (instant).2. Shift load balancer/service mesh to route 0% to canary instances.3. Scale down and terminate canary instances.4. Run reconciliation checks and execute compensation if needed (refunds, stock adjustments).5. Post‑mortem and pause further rollout.Post-release- Monitor 24–72 hours for delayed issues (reconciliation, chargebacks).- Run automated reconciliation between orders and inventory and apply fixes.- Capture lessons and update runbooks.When to choose blue-green instead- If new release requires incompatible DB changes that cannot be rolled forward incrementally AND you can accept the cost of duplicate capacity and a narrow maintenance window to cutover, then blue‑green with a read-only freeze on old environment during cutover is viable. Still require data sync strategy (CDC -> apply to green) and plan for write freeze.Final notes- Automate as much as possible: traffic shifts, flag toggles, rollback actions.- Instrument trade metrics (orders/sec, checkout success, revenue impact) and set automated alarms.- Practice this runbook with rehearsals before Black Friday.
MediumTechnical
128 practiced
Explain how you would implement staged rollouts by user cohort and by region (e.g., 1% internal users, then 5% external users in region A, then 20% region-wide). Cover user segmentation techniques, targeting mechanisms, privacy/data residency considerations, metrics to evaluate success, and concrete abort/ramp-up procedures.
Sample Answer
I would implement staged rollouts with a controlled, auditable feature-flag based progressive-delivery pipeline that supports cohort and region targeting, clear metrics, and explicit abort/ramp rules.Approach / components:- Feature flag service (LaunchDarkly/Flagsmith or internal) with targeting rules, percentage rollouts, and SDKs.- Identity & geo resolution: evaluate user identifiers (auth ID, device ID) + IP-to-region lookup or client-supplied region claim (preferable).- Cohort store: Tag users as internal (by email domain or role), beta group, or behavioral cohorts (e.g., power users) in a user attributes DB or identity provider.User segmentation techniques:- Static cohorts: based on identity attributes (employee flag, org ID).- Dynamic cohorts: server-side segments computed from analytics (last 30-day activity).- Region mapping: authoritative region claim from identity token, fallback to IP geolocation (log and surface uncertainty).Targeting mechanisms:- Use deterministic bucketing (hash(userId, featureKey) mod 100) for percentage rollouts to ensure stable experience.- Combine rules: allow internal=true -> 1%, then allow region=A and bucket <=5 -> 5% external, then region=A and bucket <=20 -> 20% region-wide.- Gate changes at API/proxy edge and client SDKs; record evaluations to logs/events.Privacy / data residency:- Avoid sending sensitive PII to third-party flag services if required by contracts. Send only pseudonymous IDs or host flag service in-region.- Ensure region decisions comply with data residency: perform geo and bucketing in-region when regulation requires (EU data stays in EU).- Store audit logs per-region and redact PII; maintain consent records for user-targeted experiments.Metrics to evaluate success:- Safety / health metrics: error rate, latency, crash rate, downstream service CPU/memory, SLI/SLOs.- Business metrics: activation, conversion, retention, revenue per user (region-specific).- Behavioral signals: usage of the new feature, fallback actions, session length.- Use dashboards and alerted thresholds (e.g., 2x baseline error rate or statistically significant negative lift).Concrete abort and ramp-up procedures:- Pre-flight: define acceptance criteria and rollback criteria in launch plan (numerical thresholds, time windows).- Automatic safeguards: server-side monitors that trigger automatic flag rollback if thresholds exceeded.- Manual abort: runbook with single-button flag-off (or kill-switch) scoped to cohort/region, paging list, communication template.- Ramp schedule example: 1. Day 0: 1% internal users (employees) — monitor 24h; criteria: no >1.5x error rate and no major drop in key business metrics. 2. Day 1-2: 5% external users in Region A — monitor 48h with tighter alerts. 3. Day 3-7: 20% region-wide — longer observation and can continue progressive increases (50%, 100%) if healthy.- Post-mortem: log all decisions, root-cause analysis, update runbooks.Trade-offs and notes:- Deterministic bucketing trades randomness for reproducibility.- In highly regulated regions prefer in-region hosting and minimal PII exchange.- Balance rollout speed with business urgency; use canary + dark launch techniques for backend changes.This design provides precise cohort control, regulatory compliance options, measurable safety nets, and repeatable abort/ramp procedures suitable for enterprise deployments.
HardTechnical
72 practiced
Create a concise emergency rollback runbook for on-call engineers and SREs that covers: immediate technical steps (how to stop/rollback the release), quick safety checks before executing rollback, communication templates for stakeholders and customers, automated tools to use, and post-rollback validation and postmortem actions.
Sample Answer
Situation: Emergency production issue caused by recent release requiring immediate rollback. This runbook is a concise step-by-step for on-call engineers, SREs, and solution architects to stop/rollback safely, communicate, validate, and follow up.Immediate safety checks (do BEFORE rollback)- Confirm scope/severity: check monitoring alerts, error rates, latency, business impact.- Verify rollback prerequisites: valid rollback artifact exists, DB migrations reversible or flagged, dependent services compatibility.- Check active writes: identify critical write windows; if non-reversible, quiesce traffic.- Confirm approvals: on-call lead + tech manager (or automated emergency policy) sign-off.Immediate technical rollback steps1. Trigger “incident” mode in incident system and assign run coordinator.2. If quick mitigation possible (feature flag/fence), flip feature flag first.3. If feature flag unavailable, perform deploy rollback: - Use CI/CD rollback job: select previous stable artifact/tag and execute rollback pipeline. - For containerized infra: update k8s Deployment to previous image tag and monitor rollout status (kubectl rollout undo or helm rollback). - For infra as code: revert to previous IaC commit and apply in blue/green or canary safe path.4. For DB changes: - If migration is backward compatible, continue. - If not, stop write traffic, restore from snapshot only if necessary (last resort).5. Route traffic: shift load-balancer to previous environment (blue/green) or decrease new canary weight to 0.Quick communication templates- Stakeholders (internal): Subject: [INCIDENT] Rollback initiated — Service X Body: We observed [impact]. Rollback to tag vX-YYYY initiated at [time]. Expected ETA: [minutes]. Owner: [name]. Next update: [time].- Customers (public status): Status: Degraded/Partial outage for Service X. We are rolling back to a previous version to restore service. Expected recovery within [ETA]. We will provide updates at [intervals].- Post-rollback update: Subject: [RESOLVED] Service X restored Body: Rollback completed at [time]. Root cause under investigation; postmortem planned.Automated tools to use- CI/CD: Jenkins/ArgoCD/GitHub Actions — use rollback pipelines with safety gates.- Orchestration: kubectl/helm for rollout/rollback; Istio/Envoy for traffic shifting.- Monitoring: Prometheus/Grafana, New Relic, Sentry for immediate verification.- Feature flags: LaunchDarkly/Unleash to toggle quickly.- ChatOps: Slack/MS Teams with incident bot to run scripted rollback jobs.Post-rollback validation- Automated health checks: run smoke tests, canary test suite, synthetic transactions.- Monitor KPIs for 2x baseline window (errors, latency, throughput).- Verify data integrity: run DB sanity checks; confirm no partial writes or duplicates.- Confirm external integrations functioning.Postmortem & follow-up actions- Within 48 hours: blameless postmortem led by owner documenting timeline, root cause, rollback decision rationale, gaps (feature flags, migration strategy).- Action items: immediate (create/enable feature flag, improve rollout gating), medium (add automated rollback playbooks in CI/CD), long-term (architecture changes to separate schema migrations, add blue/green).- Track actions in ticketing system with owners and SLA for closure.- Update runbook with lessons learned and rehearse rollback at next on-call drill.Keep this runbook accessible in the incident playbook page and ensure rollback pipelines and feature flags are tested in staging regularly.
Unlock Full Question Bank
Get access to hundreds of Deployment and Release Strategies interview questions and detailed answers.
