Release Management and Change Control Questions
Planning and governing releases: release cadence and trains, release versioning (semantic versioning, tagging, hotfix and release branches), approval gates and sign-off, go/no-go and launch-readiness decisions, release checklists, change freezes, risk-tiered change classification and pre-approved changes, change advisory and emergency-change workflows with audit trails and compliance evidence, release communication, gating releases on reliability budget, and coordinating a release across multiple services or teams, including breaking interface or schema changes with consumer deprecation timelines. The concern is the process and governance of releasing production changes, not how a deployment is executed, how pipelines are built, or how release health is measured after shipping.
After a major incident, leadership learns that your change approval process is both slow and ineffective. Redesign release governance to be faster and safer: risk tiers, which checks to automate, and the measures that would show the change worked.
Sample Answer
The idea in plain words
Leadership's complaint is that approval is both slow (changes wait) and ineffective (it does not prevent incidents). The tempting reaction after a major incident is to add more approvers, which makes the first problem worse and does little for the second. The better move is to find out from data what approval actually catches, then replace human gatekeeping on routine changes with automated checks that are cheaper and more reliable, and keep human review for the few changes where judgment matters. This is risk-based change management: control proportional to risk.
Step 1: diagnose before redesigning (first two weeks)
Pull the last 90 days of change records and incidents, and answer: how long does a change wait for approval; what share does the approval body reject; which incidents were caused by changes; and had those changes been approved. Assumed example numbers for a company with a weekly change advisory board (CAB, a meeting where proposed production changes are reviewed):
- 600 change requests in the quarter, all routed through the CAB.
- Median wait for approval 4.5 days. (The median is the middle value when waits are sorted: for waits of 1, 1, 2, 3 and 9 days the median is 2 days, however long the slowest one waits.)
- The CAB rejected 6 changes: 6 / 600 = 1 percent. The other 594 were approved (99 percent) and deployed.
- 18 incidents were caused by changes. 15 came from the 594 approved changes: 15 / 594 = 2.5 percent. The other 3 came from changes that skipped the process entirely (made directly in production, never submitted), which is why they are not among the 600 requests. Counting those 3 as well, that is 18 failures against the 603 changes the records know about, about 3 percent; the true rate is slightly lower if other direct production edits happened without causing an incident.
Reading, using the base rate (how common something is overall, before you look at a subset). 99 percent of all submitted changes were approved, and 15 of the 18 failures (83 percent) had been approved. Approval is so close to universal that "was it approved?" tells you almost nothing about whether a change will fail: a meeting that says yes to 99 changes in 100 cannot be what separates the good from the bad. The 1 percent rejection rate says the same thing, that the meeting mostly rubber-stamps. Rejected changes never run, so you cannot see how many of them would have failed, which is why the diagnosis also asks whether rejected changes later turned out to be bad (see the flip condition under Judgment). The meeting slows 100 percent of submitted changes to catch almost none. Also note the 3 failures that skipped approval: if the major incident was a change that skipped approval, the lesson is about enforcement, not about more approval.
Step 2: risk tiers
| Tier | What qualifies | Process | Assumed share |
|---|---|---|---|
| Standard | Repeatable, low-impact, has been done many times with a good record (for example, raising a retry timeout in a service config that has had 40 identical changes without incident) | Pre-approved procedure, fully automated checks, no human wait | 60 percent (360 of 600) |
| Normal | Typical code and config changes with a tested rollback | Peer review by someone who understands the service, automated risk checks, deploy when green | 32 percent (192) |
| High risk | Irreversible or wide: data migrations, schema drops, shared infrastructure, no tested rollback, large blast radius | Named reviewer panel, rollout plan, scheduled window | 8 percent (48) |
| Emergency | Fixing a live incident | Fast path with after-the-fact review | counted separately |
The shares are planning assumptions: 360 + 192 + 48 = 600. Tiers are assigned by rules (what is touched, how many dependents, whether rollback has been tested) so people cannot choose the easy one. A change moves tier when the rules say so: the same timeout edit, applied to a database parameter shared by five services with no tested rollback, is no longer Standard and becomes High risk.
Step 3: what to automate
- Tests and static checks on every change, with a policy that blocks merge on failure.
- Security and dependency scans, secret detection, and infrastructure policy checks (policy as code: rules written as code that the pipeline evaluates on every change, for example "reject any infrastructure change that opens a database port to the whole internet").
- Progressive delivery: release first to a small slice (a canary, a limited share of traffic), watch error rate and latency against the service-level objective (SLO, the reliability target), and roll back automatically if the limits are breached. For example (thresholds assumed): send 5 percent of traffic to the new version for 10 minutes and roll back at once if the error rate exceeds 1 percent or the 99th-percentile latency exceeds 800 ms against a normal 500 ms. This replaces the human guess "will this break?" with a measurement.
- Change records created automatically from the pull request and deploy, so nobody fills a form.
- Freeze and ownership checks (is this change during a freeze, does it have an owner) enforced by the pipeline.
Keep human review for irreversible actions, cross-team effects and anything the automation cannot judge.
Step 4: security and compliance without a slow gate
Enterprise security and compliance requirements still need to be met, and they are mostly satisfied by evidence rather than by meetings. Map each control to something the pipeline produces: separation of duties (no one person can both write and approve the same change; the author cannot approve their own change, enforced by repository settings), vulnerability scans, signed artifacts, and a stored record of who approved what. Agree the design with the compliance owner and auditors up front; many auditors accept well-designed automated controls, but that is for them to confirm, not for engineering to assume.
Step 5: prove it worked
Measures, taken before and after:
- Approval lead time, median and 90th percentile. The 90th percentile is the wait that 90 percent of changes are at or under: for ten waits of 1, 1, 1, 1, 2, 2, 2, 3, 4 and 9 days it is the 9th smallest, 4 days, while the median is 2 days; it shows how bad the slow tail is, which the median hides. If the Standard and Normal tiers wait roughly 0 and 4 hours and make up 92 percent of changes (360 + 192 = 552 of 600), then 92 percent of changes wait 4 hours or less, so the 90th percentile is at most about 4 hours. Before, the median alone was 4.5 days, so the 90th percentile was at least 4.5 days: the comparison is 4 hours or less against 4.5 days or more. (These tier waits are assumed.)
- Change failure rate (share of changes causing an incident, rollback or hotfix), the guard measure: speed must not be bought with failures.
- Deployment frequency (how often production deploys happen), time to restore service (the elapsed time from an incident starting to the service working again), and the share of changes auto-classified.
A caution on small numbers. 18 failures in a quarter is a small count. If failures behave like independent random events at a steady average rate (statisticians call such counts Poisson), the spread of the count equals its square root: the typical quarter-to-quarter swing around 18 is 18≈4.2, so quarters of roughly 14 to 22 are all ordinary. The square root appears because rare independent events pile up unevenly: the more events you expect, the larger the absolute swing, but it grows slower than the count itself. A drop from 18 to 15 is inside that normal swing and proves nothing. Judge the change on a rolling window (the most recent weeks or quarters, sliding forward as time passes) of at least two quarters and on failure rate per tier, not on one quarter's total.
Step 6: rollout plan
Pilot with two teams for 6 weeks. Agree in advance a stop rule with a number in it. Assumed example: the pilot teams ship about 66 changes in any two-week window and their baseline failure rate is 3 percent, so the expected count is 66×0.03≈2.0 failures, with noise of about 2.0≈1.4. Set the trigger at the baseline plus two noise units: 2.0+2×1.4≈4.8, rounded to 5. If any rolling two-week window in the pilot reaches 5 or more change-caused failures, return those teams to the old process and investigate. Fewer than 5 is within the normal swing and is not by itself a reason to stop. Then expand team by team.
Judgment
I would replace the CAB's per-change review with tiering and automation, and keep a smaller board only for the high-risk tier. What would flip it: if diagnosis shows the CAB does catch real defects (many rejected changes that later proved to be bad), keep that review and speed it up instead.
Common mistakes
- Adding approvers after an incident without checking whether approval was the failure.
- Declaring success from one quarter's incident count.
- Automating a check without a tested rollback behind it.
- Letting teams pick their own tier.
How would you gate production promotion on how much of the service's reliability budget has been consumed, for example blocking when the last 24 hours burned more than 5 percent? Describe the rule, who can override it, and how it fits into release automation.
Sample Answer
Direct answer
Add a policy check stage in the release pipeline that reads the service's error budget burn from the metrics system and refuses to promote a change to production when the last 24 hours consumed more than 5 percent of the monthly budget, or when the budget is already spent. Define the rule in numbers (below), let only named people override it with a recorded reason and an expiry, and exempt reliability fixes, because blocking the fix for an outage is backwards.
Terms: a service level objective (SLO) is a reliability target, for example "99.9 percent of requests succeed over 30 days". The error budget is the amount of failure that target allows. Burn is how much of that budget has been used. It is read two ways below: as a share (percent of the monthly budget used, a total) and as a pace (how fast it is being used, as a multiple of the steady rate the SLO can afford, a rate). A promotion is moving a build to the next environment or to a wider share of production traffic. A canary is the first small share of live traffic that a new build serves while the rest stays on the old build; canary analysis compares the canary's error rate with the old build's. An incident commander is the person coordinating the response to an active incident. An audit log is a record of decisions that cannot be edited afterwards. A service level indicator (SLI) is the measurement the target is judged on, for example the share of requests that succeed. A runbook is the written procedure for operating a service: how to tell it is unhealthy, how to roll back, and who to call.
The rule in numbers
Budget for a window is the allowed bad events: budget=(1−SLO)×N where N is the total events (requests) in the 30-day window. The burn over the last 24 hours as a share of that budget is:
burn24h=budgetbad events in the last 24 hoursWorked example (assumed traffic)
Assumptions: SLO 99.9 percent over 30 days; the service handles 100,000,000 requests in 30 days, evenly spread.
- Budget: 0.001×100,000,000=100,000 bad requests.
- Requests per day: 100,000,000/30=3,333,333.
- The 5 percent line: 0.05×100,000=5,000 bad requests in 24 hours.
- As an error rate (a different unit: percent of requests, not percent of the budget): 5,000/3,333,333=0.15%, which is 1.5 times the 0.1 percent the SLO allows. That ratio of 1.5 is the pace.
- A sustainable day uses 1/30=3.33% of the budget, and 5%/3.33%=1.5. If every day burned 5 percent, the budget would be gone in 100/5=20 days, ten days before the window ends.
So the rule blocks promotion whenever the service has been failing at more than 1.5 times the pace the SLO can afford, a clear "stop adding change" signal that is milder than a page (an alert that wakes the on-call engineer, the person currently responsible for answering problems with the service).
Rule design (one clause per row)
| Clause | Behaviour |
|---|---|
| Signal | The same indicator the SLO uses (for example the share of successful requests), not a separate metric |
| Block condition | burn24h>5% of the 30-day budget, or remaining budget at or below zero |
| Minimum volume | Evaluate only if the window had enough traffic; otherwise use a longer window or a manual check |
| Exempt lane | Rollbacks, reliability fixes and security patches are never blocked by this rule; they have their own checks |
| Metrics unavailable | Treat as unknown: block ordinary promotions with an override path, do not block rollbacks |
| Where it runs | Before each promotion step (to canary, to 25 percent, to 100 percent), not only at the start |
One gate decision, end to end (same assumptions)
The function below is what the pipeline step does with the numbers it reads. Minimum volume here means the check only blocks when the window held enough requests to be meaningful; it is omitted from the code to keep the example short. It prints four invented readings, one for each of four different services:
SLO = 0.999
REQUESTS_30D = 100_000_000
BUDGET = round((1 - SLO) * REQUESTS_30D) # allowed bad requests in 30 days
REQUESTS_24H = REQUESTS_30D / 30
LIMIT = 0.05 # block above 5 percent of the monthly budget in 24 hours
def gate(bad_24h, bad_30d):
if bad_24h is None:
return "UNKNOWN: block ordinary promotions, never block rollbacks"
share = bad_24h / BUDGET
pace = (bad_24h / REQUESTS_24H) / (1 - SLO)
numbers = f"{share:.1%} of budget in 24h, {pace:.2f}x the sustainable pace"
if bad_30d >= BUDGET:
return f"BLOCK: budget spent ({bad_30d} of {BUDGET}); {numbers}"
if share > LIMIT:
return f"BLOCK: {numbers}"
return f"ALLOW: {numbers}"
print("budget:", BUDGET, "bad requests; 5% line:", int(LIMIT * BUDGET))
for label, bad_24h, bad_30d in [
("orders", 3_200, 41_000),
("search", 6_800, 47_800),
("billing", 1_000, 100_500),
("reports", None, None),
]:
print(f"{label:<10}{gate(bad_24h, bad_30d)}")
Output
budget: 100000 bad requests; 5% line: 5000
orders ALLOW: 3.2% of budget in 24h, 0.96x the sustainable pace
search BLOCK: 6.8% of budget in 24h, 2.04x the sustainable pace
billing BLOCK: budget spent (100500 of 100000); 1.0% of budget in 24h, 0.30x the sustainable pace
reports UNKNOWN: block ordinary promotions, never block rollbacks
The orders service: 3,200 bad requests is 3.2 percent of the 100,000 budget, under the 5 percent line, so the promotion goes ahead. The search service: 6,800 is 6.8 percent, which is 2.04 times the sustainable pace, so it is blocked and the figures go to the release channel. The billing service: the last 24 hours burned only 1.0 percent, but 100,500 bad requests over the month have already passed the whole 100,000 budget, so ordinary promotions stop. The reports service: the metrics store (the system that holds the success and failure counts) did not answer, so the result is unknown and follows the table above.
Why a minimum volume clause matters
For a service with only 10,000 requests in 30 days, the budget is 0.001×10,000=10 bad requests and 5 percent of it is 0.5, so a single failed request blocks all releases. The gate must either use a longer window for low-traffic services or require a minimum number of requests before it blocks.
Who can override
- The service owner and the on-call lead (or incident commander during an incident), and not the author of the change alone, so that an override is a second person's decision.
- Every override records who, why, the ticket or incident, and an expiry (for example 4 hours or the single promotion). The audit log entry is written by the pipeline, not typed by the person.
- Overrides are reviewed weekly: a service that needs them repeatedly has a reliability problem or a badly set threshold, and the review decides which.
Which checks are automatic and which need a person
Anything that is a number, a lookup or a comparison runs in the pipeline on every promotion step. Anything that is a judgment about intent or risk goes to a named human, and the pipeline records who decided.
| Check | Decided by | Why |
|---|---|---|
| Burn share and pace against the threshold, and remaining budget | Pipeline, automatically | Pure arithmetic on the SLI, must give the same answer every time |
| Enough traffic in the window, and metrics fresh enough to trust | Pipeline, automatically | A data-quality test, not an opinion |
| Canary error rate against the baseline | Pipeline, automatically | Repeats at each wider traffic step |
| Linked runbook present and current (next section) | Pipeline checks the fields; the owning team judges the content | A script can see a date and a link, not whether the steps are right |
| Whether a change is a rollback or reliability fix | Pipeline reads the label on the change; a named second person confirms any non-rollback fix label | Stops "fix" becoming a label everyone attaches to skip the gate |
| Override of a block | A human: service owner or on-call lead, recorded with reason and expiry | Accepting risk is a decision, not a computation |
| Changing the threshold or the rule | Humans, as a reviewed change to the rule file | The policy itself is released like code |
A linked, current runbook as a release precondition
A production promotion also requires that the release record links to a runbook for the service, and that the runbook is current. Current means: the link resolves, the page names the owning team and the on-call rotation, it describes how to roll back and which SLI dashboard to read, and it was reviewed within an agreed period (90 days is an example, set by the organisation) or since the last major architecture change. The pipeline can check exactly those facts: link, owner field, rollback section present, review date. It cannot judge whether the steps are good, so quality is checked by the owning team when the runbook changes and tested in incident reviews and drills. A missing or stale runbook blocks ordinary promotions in the same way as the burn gate (same override path, same expiry), and rollbacks are never blocked by it. The reason for tying the two together: when the burn gate blocks or a canary fails, the person who is woken needs the rollback steps in front of them, not a search through chat history.
How it fits into release automation
- The pipeline reaches the production promotion stage.
- It calls a small policy service (a small internal web service that holds the rules and answers "may this proceed?") that queries the metrics store for the SLO window and returns allow, block or unknown, with the numbers used, and in the same call confirms that a linked, current runbook exists. In practice the pipeline step is a single call whose answer decides whether the next stage runs, like the function above.
- Allow continues; block stops the pipeline and posts the burn figure and a link to the dashboard in the release channel; unknown follows the table above.
- The decision, inputs and the policy version are written to the audit log.
- Each wider traffic step repeats the check, and the canary's own error rate is compared with the baseline so a change that causes the burn is stopped by its own canary, not only by the budget gate.
Judgment and limits
The 5 percent figure is a starting point: tune it with history by replaying past months and counting how often it would have blocked. Too tight blocks healthy releases; too loose never fires. A budget gate is a blunt tool: it does not know whether the burn came from your last release, a dependency, or a traffic spike, so keep it alongside canary analysis rather than instead of it. Slow-burning problems are caught by a longer-window check; fast ones by the canary.
Common mistakes
- Blocking fixes and rollbacks.
- Using a different measure from the SLO, so the gate and the target disagree.
- No override path, which teaches teams to work around the gate.
- Applying the rule only at the start of a rollout.
Tell me about a release you were responsible for that went poorly. What was your role, how did you coordinate and communicate, how did you limit customer impact, and what did you change afterwards?
Sample Answer
The idea in plain words
A "release that went poorly" question is not testing whether you have never failed. It tests four things: that you owned the outcome, that you can run a calm, organised response when production is hurting (coordination and communication), that you put the customer's pain ahead of your pride (limiting impact), and that you turned the failure into a change in how the team works. Tell it with the STAR shape: Situation, Task, Action, Result. Spend most of the time on Action and on what changed afterwards.
The story below is a worked skeleton. Its facts are invented for illustration; substitute your own real release, with your own real numbers, and keep only the structure.
Situation and role
"I was the release owner (the person accountable for getting one change safely to production) for a change to a nightly-and-hourly data pipeline: a transformation step that converts order amounts to a single reporting currency. Downstream, a revenue dashboard and an export that finance teams at customer companies use read the pipeline's output table. My role was to plan the release, run the go/no-go call (the checkpoint just before release where each owner says go or hold), and coordinate the response if something broke. I was not the author of the code; that mattered, because my job was to check the process, not defend the change."
What went wrong
The new transformation applied the currency conversion twice for orders that were already in the reporting currency. Amounts are stored in the currency's minor unit, the smallest coin (cents for dollars), so 20.00 is stored as 2000. For a simple illustration, suppose the region's rate was 2.0: an order already in the reporting currency should have been left alone at 2000, but the new code multiplied it by the rate again and stored 4000, which is why that region's totals looked about double. (The rate and amounts are illustrative.) Unit tests passed, because the test fixtures only contained orders in one currency. The pre-release validation compared row counts between old and new output, and the row counts matched, because the bug changed values, not the number of rows.
Illustrative timeline (all times in one time zone, same day):
| Time | Event |
|---|---|
| 14:00 | Change deployed; the scheduler (the component that starts the job each hour) is set to run it at five past each hour |
| 14:05 | First run starts; it finishes at 14:12 and publishes to the output table |
| 14:50 | A finance analyst at a customer reports that totals look about double for one region |
| 14:58 | I pause the pipeline scheduler so no further bad runs publish |
| 15:20 | Previous code version redeployed |
| 15:45 | The one bad hour is reprocessed with the old code and the table is corrected |
| 16:10 | Validation checks pass on the corrected hour; customers told it is resolved |
Arithmetic from the table: wrong numbers were visible from 14:12 to 15:45, which is 93 minutes (14:12 to 15:12 is 60, 15:12 to 15:45 is 33). Detection took 38 minutes from the first publish (14:12 to 14:50). Because I paused the scheduler at 14:58, the 15:05 run never happened, so exactly one hourly batch was bad. Had I waited for the rollback at 15:20, a second bad batch would have published at 15:12.
Which number to quote: the customer-impact figure is the 93 minutes during which wrong numbers were visible to customers, and the fact that only one hourly batch was affected. The 38 minutes is the time to detect, an internal figure that belongs with the follow-up actions.
How I coordinated and communicated
- Declared it and named roles within minutes. I opened an incident channel, took the incident commander role myself for the first ten minutes (the incident commander coordinates and decides; they do not debug), then handed it to the on-call engineer so I could stay on communication. I assigned one person to investigate, one person to keep a running log of times and decisions, and one support contact for customer messages.
- Stopped the bleeding before understanding the cause. Pausing the scheduler was a decision I could make with partial information, and it was reversible. Fixing the transformation was not yet possible, because we did not yet know the cause.
- Chose rollback over fix-forward. Rollback means redeploying the previous version; fix-forward means writing and shipping a new corrected change instead. The previous version was known good and redeploying it took one pipeline run of effort; debugging the double conversion live would have taken an unknown time.
- Communicated on a fixed rhythm. Internal updates every 20 minutes to the channel (what we know, what we are doing, when the next update is), even when the update was "no change". Support got a short customer-facing statement at 15:05 so they were not improvising answers. It read roughly: "We are aware that revenue totals for one region show about double for the 14:00 hour. We have paused further updates; earlier hours are correct. Next update at 15:25." A 20-minute internal update read roughly: "Known: one hourly batch is wrong, scheduler paused, no new bad data. Doing: previous version being redeployed, ETA 15:20. Next update at 15:20."
- Told customers directly. Customers who consume the export got an email saying which hour of data was affected, that it was corrected, and what to do if they had already downloaded it.
How I limited customer impact
- Stop further bad data (pause the schedule) before anything else.
- Mark the data, not just the code. Rolling back code does not repair rows that were already written. The bad hour had to be reprocessed with the old logic, and the dashboard showed a banner on the affected hour until the correction landed.
- Reach the people who already acted on the numbers. Exports that were already downloaded are outside your control; the email told them exactly which rows to discard.
What I changed afterwards
Each change was tied to a specific gap that the timeline exposed, with an owner and a date:
| Gap | Change |
|---|---|
| Fixtures held one currency | Test data now includes every currency and a deliberate unit-edge case |
| Row counts matched but values were wrong | Validation compares sums and a sample of values per currency between old and new output, not only counts |
| The change went to all data at once | Releases of pipeline changes first run in shadow mode (new logic runs on the same input, writes to a side table, nothing downstream reads it) and are compared before cutover (the moment real consumers start reading the new logic) |
| 38 minutes to detect | An automated check on the published batch compares the revenue total with the previous hour and the same hour a week earlier, and blocks publication on a large difference |
| Pausing was ad hoc | The publish step has a kill switch (a control that stops publishing at once, without a deploy), documented in the runbook (the written step-by-step for an operational task) |
| Releases at peak finance reporting hours | Pipeline releases avoid the hours when customers pull exports |
I also ran a blameless review (a post-incident review that studies the process and system conditions rather than assigning fault to a person) within the week and shared the written summary with the teams that consume the pipeline.
What strong and weak versions sound like
- A strong answer owns the decision points ("I should have required a shadow run for a transformation change") and shows a changed process, not only an apology.
- A weak answer blames a person, a vendor or "a lack of time", gives a story where nothing was your decision, or ends at "we rolled back" with no learning.
- Choose a failure that was real and recoverable. A story with a made-up villain, or one so large it suggests you should not have been trusted with the release, both work against you.
- Do not invent precision. If you do not remember a number, say "roughly" and give the order of magnitude.
Draft a branching and release policy that avoids long-lived branches but still supports emergency hotfixes. What rules would govern what lands on a release branch, who approves a hotfix, and how fixes are merged back?
Sample Answer
The idea in plain words
A long-lived branch is a copy of the code that drifts away from the main line for weeks or months, so every merge becomes a painful, risky event. The way out is trunk-based development: everyone merges small changes into one main branch (the trunk) at least daily, the trunk is always releasable, and unfinished work is hidden behind feature flags (switches that turn code paths on or off without a new deployment). A release branch is then not a place where work happens. It is a short-lived pin on a known-good commit, used only when you must fix a released version without picking up newer work from main. A hotfix is an urgent fix to production that skips the normal queue.
The single rule that keeps this safe is main first: a fix lands on main, and only then is a copy of it (a cherry-pick, meaning one commit re-applied on another branch) placed on the release branch. If every fix flows main to release, there is nothing to merge back and the two lines cannot diverge.
The policy, as rules
Branches
mainis protected (a setting on the code host that blocks direct pushes and force pushes): pull requests (PRs) only, required automated checks (continuous integration, CI) must pass, at least one approval from a code owner (a person or team listed as responsible for that part of the code, whose review is required for changes to it).- Feature branches live at most 2 working days. Anything longer is split up or put behind a flag. (The 2-day figure is a team default to tune, not a standard.)
- A release branch
release/MAJOR.MINOR(for examplerelease/2.7) is cut from a specific commit on main only when a release needs stabilising or a version must be supported after main moves on. It is deleted when that version leaves support. For a hosted service deployed from main many times a day, you may need no release branches at all, only tags: to release or hotfix you tag the exact commit on main you deployed, and if you need to hotfix an older deploy you can branch from that tag at that moment. - Versions follow semantic versioning: MAJOR.MINOR.PATCH, where MAJOR means a breaking change, MINOR a backward-compatible feature, PATCH a backward-compatible fix. Every release and hotfix is a signed tag (a tag is a permanent label on one commit, such as
v2.7.1; signing it proves who created it and needs a signing key set up first, otherwise git stops with an error such asgpg failed to sign the data); a hotfix increments PATCH only (2.7.0 to 2.7.1).
What may land on a release branch
- Only fixes for a defect that blocks the release or harms production: data loss or corruption, security vulnerability, outage, or a legal or contractual breach. Never features, refactors or dependency upgrades unrelated to the fix.
- Only commits that already exist on main (cherry-picked with
git cherry-pick -x, where-xappends a "cherry picked from commit ..." line so the origin is recorded). Exception: if the code on main has changed so much that the fix cannot be written there first, the fix is written on the release branch and a PR to main is opened in the same hour, with a ticket that blocks closing the incident. - Each release-branch PR states which defect it fixes, the risk, and how it was tested, including a regression test (a test that fails before the fix and passes after, so the bug cannot quietly return).
Who approves a hotfix
- Author: any engineer on the owning team.
- Approver: one code owner of the affected area who did not write the change. For a declared incident, the incident commander (the person coordinating the response) or the on-call lead may approve in place of the code owner when no code owner can respond within 15 minutes. That approval is reviewed afterwards by a code owner within 1 business day.
- Automated checks (build, unit tests, the tests covering the fix, a security scan) still run. Only the human wait is shortened, not the machine checks.
How fixes are merged back
- Normal path: nothing to merge, because the fix was on main first.
- Exception path: the PR from the release fix to main is opened within the hour (as above) and merged within 24 hours. A scheduled check runs
git cherry main release/2.7. Read the command as "upstream first, then the branch to inspect": it lists the commits that are onrelease/2.7and not onmain, and for each one asks whether main already has an equivalent change. Patch equivalence means git compares the content of the change (the lines added and removed), not the commit id, so a cherry-picked copy with a different id still counts as the same change. A line starting with+means a change that is on the release branch but has no equivalent on main; a line starting with-means main already has an equivalent. Git prints the full 40-character commit id after the sign; the examples here shorten it. Any+line older than 24 hours pages the owning team.
Worked scenario (assumed numbers, not a real system)
Assume version 2.7.0 was cut from main at commit a1c3 on a Monday and shipped Tuesday. Main has since gained 40 commits.
- Thursday 10:00: a bug corrupts exported invoices. The owning team confirms it is a data-integrity defect, so it qualifies for the release branch.
- 10:40: the author opens a PR to main with the fix (commit
f9d2) and a regression test. A code owner approves at 11:05, CI passes, it merges. - 11:15:
git switch release/2.7thengit cherry-pick -x f9d2creates commitc7e0on the release branch. Flags:switchchanges branch;-xrecords the origin. - 11:25: a release-branch PR runs CI and the release test suite, a code owner approves, and the tag
v2.7.1is created withgit tag -s v2.7.1 -m "Fix invoice export corruption"(-ssigns the tag with the releaser's key, which must already be configured as described above;-msets the message). - 11:40: v2.7.1 is deployed.
git branch -r --contains f9d2confirms the fix is reachable fromorigin/main;git cherry main release/2.7prints- c7e0, the minus sign meaning an equivalent patch already exists on main. Nothing is left to merge back.
Total time from diagnosis to deploy in this sketch is 100 minutes (10:00 to 11:40), and the only human gates were one code-owner approval on main and one on the release branch.
Judgment and what would change it
I would adopt trunk-based development with tags and flags for a continuously deployed service, and add short release branches only for versions customers run themselves (packaged or on-premise software), where you cannot force every customer onto the latest. The cost of release branches is carrying fixes to more than one place; the main-first rule keeps that cost to a cherry-pick. Measures that show the policy works: age of every release branch, count of + lines from git cherry, time from hotfix approval to deploy, and the share of hotfixes that needed the exception path (a high share means main is too unstable to take fixes first).
Common mistakes
- Fixing on the release branch first "because it is faster" and forgetting main, so the bug returns in the next release.
- Letting release branches accept "just one small feature", which turns them back into long-lived branches.
- Merging the whole release branch back into main instead of cherry-picking, which drags unrelated release-only commits along.
- Giving hotfixes no after-the-fact review, so the emergency path becomes the normal path.
What are release gates? For a regulated payment service, which gates would you automate and which would need a human approver, and why is each one blocking or advisory?
Sample Answer
The idea in plain words
A release gate is a checkpoint a change must pass before it moves to the next stage (merge, pre-production, production, wider rollout). A gate is blocking when failing it stops the release until the problem is fixed or a named person formally accepts the risk. It is advisory when failing it produces a warning and a record but the release can continue. The rule of thumb: automate every gate whose answer is a repeatable fact (tests pass, no known critical vulnerability), and keep a human for gates that need judgement about risk, business timing or independence (separation of duties). In a regulated payment service, each gate also leaves evidence an auditor can read later.
Terms: separation of duties means the person who writes a change cannot be the only person who approves it. PCI DSS (Payment Card Industry Data Security Standard) is the security standard for systems that handle card data; it expects documented change control, so check the current version of its text and your assessor's reading for the exact requirements. Fail closed means if a gate cannot run (the scanner is down), it blocks rather than passes.
Gates for the payment service
Assume the service takes card payments, calls an external acquirer (the bank that processes the card), and stores no raw card numbers.
| # | Gate | Automated or human | Blocking or advisory | Why |
|---|---|---|---|---|
| 1 | Unit, integration and payment-flow tests (authorise reserves the customer's money, capture actually takes it, refund gives it back; plus declined card and timeout cases) against a sandbox acquirer (the bank's test endpoint, which behaves like the real one but moves no money) | Automated | Blocking | A failure means money moves wrongly; the result is a plain fact |
| 2 | Secret scan and dependency scan: any critical or high known vulnerability, or any committed credential | Automated | Blocking for critical and high; advisory for medium and low with a ticket and a deadline | Known exploitable flaws in a payment path are not a judgement call; lower severities are tracked so they do not stall every release |
| 3 | Independent code review by someone other than the author | Human | Blocking | Separation of duties; enforced by branch rules so it cannot be skipped |
| 4 | Security review for changes touching the card data flow, authentication, cryptography or tokenisation (replacing card numbers with random stand-in values so other systems never hold the real number) | Human, only when the change touches those areas | Blocking for those changes | Judgement about design, not only about known patterns; routed by file ownership rules so unrelated changes never queue for it |
| 5 | Performance and latency check against the service-level objective (SLO, the agreed reliability target) in pre-production | Automated | Advisory when within the error budget (the amount of failure the SLO allows, worked in numbers just below); blocking if it would breach the SLO | Slow checkout loses payments, but a small drift should not stop a fix |
| 6 | Production approval by the service owner or change approver, with a change record | Human | Blocking for normal changes; replaced by pre-approval for catalogued low-risk standard changes (routine change types with a known procedure that were approved in advance) | Accountability and the audit trail; no change reaches production without a named accountable person |
| 7 | Canary health (a canary is the new version serving only a small slice of live traffic while the rest stays on the old one): error rate, authorisation success rate and latency of a small traffic slice compared with the old version | Automated | Blocking, with automatic rollback | Only real traffic reveals some failures; the decision is mechanical |
| 8 | Evidence capture: links to test results, scan results, reviewer identity and deploy time stored in an append-only record (entries can be added but never changed) | Automated | Blocking: a release with missing evidence is treated as failed | An auditor needs proof, and gaps found later cannot be repaired |
That is eight gates: five fully automated (1, 2, 5, 7, 8), two human (3, 4), and one human approval (6) that can be pre-authorised for standard changes.
Gate 5 in numbers (assumed)
Suppose the SLO is that 99.9 percent of payment requests succeed within 1 second over 30 days, so the error budget is the 0.1 percent that may fail or be slow. With 3,000,000 requests in the window that is 3,000 bad requests; 1,200 are already spent, leaving 1,800. A pre-production load test predicts how many extra bad requests the new version would add over the 1,500,000 requests still expected:
slo = 0.999 # 99.9 percent of payment requests succeed within 1 second
requests_30d = 3_000_000
budget = round((1 - slo) * requests_30d) # failed or slow requests the SLO allows
used = 1_200 # already spent this window
remaining = budget - used
requests_left = 1_500_000 # requests still expected this window
print("budget:", budget, "used:", used, "remaining:", remaining)
for label, extra_rate in [("small drift", 0.0005), ("large regression", 0.0015)]:
extra = round(extra_rate * requests_left)
verdict = "advisory (fits)" if extra <= remaining else "blocking (would breach the SLO)"
print(f"{label:<17}{extra_rate:.2%} slower -> {extra} extra bad requests: {verdict}")
Output
budget: 3000 used: 1200 remaining: 1800
small drift 0.05% slower -> 750 extra bad requests: advisory (fits)
large regression 0.15% slower -> 2250 extra bad requests: blocking (would breach the SLO)
A 0.05 percent drift adds 750, which fits in the 1,800 left, so the gate warns and the release goes on. A 0.15 percent regression adds 2,250, more than the 1,800 left, so the gate blocks.
Worked example
A release adds a retry on acquirer timeouts and upgrades a logging library.
- Gate 1 passes. Gate 2 reports one medium-severity vulnerability in the logging library and no critical or high ones: advisory, so the release proceeds, a ticket is opened with a deadline, and the finding is recorded in the evidence.
- Gate 3 passes after a second engineer reviews it.
- Gate 4 is triggered because the retry touches the payment authorisation flow. The security reviewer asks what happens if a retry follows a timeout that actually succeeded, which could charge a customer twice. The team adds an idempotency key (a unique identifier sent with the request so repeated requests are treated as one; idempotency is the property that doing something twice has the same effect as doing it once) and the reviewer approves. That is the value of a human gate: no scanner would have noticed it.
- Gate 6 is signed by the service owner. Gate 7 shows the canary's authorisation success rate matches the old version, so the rollout widens. Gate 8 stores the full record.
Judgement calls
- Fail closed. If the vulnerability scanner or evidence store is down, the gate blocks. Provide a documented break-glass path (a pre-agreed emergency override by two named people, time-limited and reviewed afterwards) so an outage does not make fixing production impossible.
- Advisory gates need an owner and a deadline. An advisory result nobody reads becomes noise. Review advisory findings on a schedule, and promote a gate to blocking if the same warning keeps recurring.
- Do not put humans where machines are better. A person eyeballing test results adds delay and no safety. Keep human time for gates 3, 4 and 6.
- Keep gates few and fast. Every gate adds waiting time, so remove any that has never caught anything, and track how many releases each gate stopped.
This table is an illustration of the reasoning for a typical payment service; a real programme must also cover the regime's own requirements (for example data retention of evidence, segregation of environments and who may deploy), the organization's risk appetite and the assessor's interpretation.
Unlock Full Question Bank
Get access to all 21 Release Management and Change Control interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.