Prioritization and Trade-Off Decisions Questions

Decide the split with a reliability budget, not a feeling. An SLO (service level objective) is a target such as 99.9% of requests succeeding. The error budget is the unreliability it allows. Google's SRE book defines it as the gap between the SLO and measured uptime, and says releases can continue while the SLOs are met and are temporarily halted when frequent violations spend the budget. Google's example error budget policy goes further: when a service exhausts its budget over a four-week window, teams halt changes other than P0 issues (the most severe, drop-everything incidents) or security fixes until it is back within the SLO, and disagreements about it escalate to the CTO.

Worked example (illustrative). A 99.9% SLO over a 90-day quarter allows 90 x 24 x 60 x 0.001 = 129.6 minutes of failure. That is the error budget, measured in minutes of downtime. A team of 10 has 10 x 13 = 130 engineer-weeks of capacity in the quarter. These are two separate quantities (minutes of failure allowed, and engineer time available): the budget tells you WHEN to move the split, the capacity tells you WHAT moving it costs.

• Default split: 20% reliability (26 engineer-weeks), 80% features. This is a starting point to tune: it keeps feature delivery dominant while funding steady maintenance.
• Trigger: less than 25% of the budget remaining (under 32.4 of the 129.6 minutes), or a repeat incident (the same root cause causing a second outage), or a burn rate well above 1 (a threshold I would tune, for example 1.5) because that projects exhaustion before the quarter ends even while plenty of budget remains. A quarter left leaves time to react before the budget hits zero. Then move to 40% reliability (52 engineer-weeks).
• Budget exhausted: freeze feature releases except urgent fixes until reliability work restores it.

Burn rate is how fast the budget is being spent relative to the time passing. Example: 40 minutes of failure in the first 15 days of the 90 uses 40 / 129.6 = 31% of the budget in 17% of the quarter, a burn rate of about 1.9 (spending nearly twice as fast as the budget allows). At that pace the budget runs out around day 49 (129.6 / (40/15) = 48.6 days), so it is a signal to move to 40% now, even though 89.6 minutes (69%) of the budget remains and the 25% remaining-budget trigger has not fired: that is why burn rate is a trigger of its own and not only a piece of evidence.

Evidence that moves the split: burn rate, repeat-incident count, on-call hours per week (on-call is the rotation of engineers who respond to alerts), customer-visible impact (tickets, churn at risk), and the cost of the incident backlog.

Showing non-engineers. One chart, plus a two-line trade. The chart: a line of budget remaining from 129.6 minutes at day 0, falling to 89.6 at day 15, with a horizontal line at 32.4 marking the 25% trigger. The trade: "Moving 20% to 40% takes 26 more engineer-weeks, which pushes feature X (about 14 engineer-weeks) and feature Y (about 12) to next quarter, and cuts expected downtime." Those two feature sizes are illustrative; the 26 is the arithmetic: 52 - 26.

The same method in other situations.

• Fragile ETL (extract-transform-load, the pipeline that copies and reshapes data) vs a churn dashboard: if the dashboard reads the fragile pipeline, fix the pipeline first since the dashboard would be wrong.
• Automation vs incident backlog: 5 incidents a month at 2 hours is 10 hours a month, 120 hours a year; automation costing 3 engineer-weeks (120 hours at 40 hours per week) pays back in about a year. The incident fix that removes the repeating cause ranks ahead: it stops the incidents (and the customer impact) entirely, while automation only makes handling them cheaper.
• Network incident vs project work: the budget does not care about the cause. Say a network outage burns 45 minutes of the 129.6; that is 35% of the budget, the remaining budget falls to 84.6 minutes, and the same rules apply as for a code bug.

What flips it: a launch tied to revenue where the budget is healthy.

Direct answer. Triage on three questions asked in the same order every time: how bad is the harm, how many are hit, and is there a workaround. Fix now means harm is severe or growing; schedule means real but contained; defer means low harm, narrow reach or an easy workaround. Every deferred bug still gets a reply and a re-review date.

Design

• Severity (Sev1 to Sev4): Sev1 is data loss, security or a broken core flow; Sev2 is a major feature broken; Sev3 is degraded with a workaround; Sev4 is cosmetic.
• Reach tier (1 to 3): 1 is one user or team, 2 is several teams, 3 is a large share of active users.
• Escalation likelihood (0 to 2): 0 none, 1 a paying customer or key internal team is waiting, 2 an executive or contract date is involved.
• Score = severity weight (Sev1=8, Sev2=4, Sev3=2, Sev4=1) x reach tier x (1 + 0.5 x escalation). The weights are a starting design to tune, not a standard.

A Sev1 is fixed now regardless of score, and the score ranks everything else. Cutoffs, set before scoring so nobody picks them to fit a favourite: a score of 20 or more is fix now, 5 to 19 is schedule, below 5 is defer. Checked against the table: 36 is fix now, 12, 8 and 6 are schedule, 2 is defer. Automated ranking of operational issues works the same way: a script computes the score from alert severity, count of affected accounts and open escalations, and a human reviews only the top of the list and anything that jumps.

The person whose bug is deferred. Reply within a day with the decision and the reason in their terms, give a workaround, state what would raise it (more reports, a new customer, an escalation) and a re-review date, and leave it visible as deferred rather than closed. Silence, not the no, is what damages trust.

Pitfall. A queue that only grows hides the real defer decision. Review the deferred list monthly and close with an honest "will not fix" when it stays low.

Short answer

First work out whether containment (a quick measure that reduces the damage now without removing the cause) pays for itself while the systemic fix (the permanent repair of the root cause) is being built. Then, when each incident is expensive, do both: contain now, fix systemically, and stop the containment on a fixed date (the sunset). The systemic fix is the destination, and containment is bridge funding (temporary money to cover the gap until the permanent fix lands) with an end date.

Step 1: Compute the break-even for containment

Illustrative assumption (the question gives none): 20 incidents a month. An 80% cut avoids 20 x 0.8 = 16 incidents a month. Containment costs $50,000 a month, so it pays for itself if each incident costs more than $50,000 / 16 = $3,125.

Now suppose each incident costs $10,000 (on-call time, customer credits, lost work). Assume the systemic fix costs $300,000, lands at month 12, and then ends the incidents. "Total" below is incident cost plus spend.

The 24-month column: containment only keeps costing (4 x 24 x $10,000 + 24 x $50,000 = $2,160,000), the fix alone leaves a full year of unmitigated incidents, and both is cheapest at $1,380,000 because the incidents and the spend stop after the fix. Over the first 12 months, doing both costs $300,000 more than containment alone ($1,380,000 versus $1,080,000), and that $300,000 buys the permanent end of the problem. Comparing spend only, the fix is $300,000 once while containment is $600,000 a year and recurring, so the fix should start now; the question is whether to also pay for the bridge. The 12-month totals favour containment alone ($1,080,000 versus $1,380,000 for both), because the fix has not yet paid back; the 24-month totals favour doing both ($1,380,000 versus $2,160,000), because the fix ends the incidents and the containment spend. So doing both is right when the horizon is longer than the fix date and incidents cost this much.

Step 2: Do both when

• the break-even holds with margin (incidents are costly, as above);
• the systemic fix takes months to land, so incidents continue meanwhile;
• the bridge has a sunset. If the fix lands in month 9 instead, containment runs 9 x $50,000 = $450,000, the fix costs $300,000, so spend is $750,000 (compare that with the $900,000 of spend when it runs the full 12 months). In total terms: incidents cost 9 x 4 x $10,000 = $360,000, so the total is $360,000 + $750,000 = $1,110,000 versus $1,380,000. Compare spend with spend and totals with totals.

Step 3: Choose one only when

• Systemic only: incidents are rare or cheap (below the break-even), or customers can tolerate the pain.
• Containment only: the component will be retired soon, so a permanent fix would be wasted.

Risks

Containment can relieve the pressure and the systemic work then slips. Protect it: fund both from the start, publish the sunset date, and track a milestone for the fix.

The same logic for recurring delivery problems

For repeated release failures, a quick win (a checklist, an automated rollback) is containment and a rebuilt pipeline is the systemic fix. For "a robust pipeline in three months versus one dashboard now", ship the dashboard first on a manual data pull with an expiry date, because decisions are being made blind today, then build the pipeline so the dashboard stops needing manual work. Technical debt (shortcuts that make later work slower) is what a manual dashboard becomes if the expiry date is ignored.

What would change the call: a measured per-incident cost below $3,125, or the fix taking far longer than twelve months.

Direct answer. Work the outage first, give the executive an immediate heads-up and a realistic time, hold the minor UI bug for the normal queue, and schedule the policy update with a protected slot. Sort by urgency (how soon it hurts) and importance (how much it matters), the Eisenhower idea: important and urgent first, important and not urgent scheduled, urgent but less important handled or delegated, and the rest dropped or batched.

Order and reasoning

1. Outage affecting a small customer segment. Users are harmed now and it may grow. Acknowledge it, assess scope in minutes, and fix or mitigate. If someone else is on the incident, support them instead of duplicating effort.
2. Finance executive's data request due end of day. Fixed deadline and a senior stakeholder. Reply early: confirm what you will deliver and by when; if the outage slows you, say so and offer a partial answer or a later time rather than going silent.
3. Policy update due next week. Important but not urgent today: block a calendar slot now (for example tomorrow morning), so it does not become today's emergency.
4. Minor UI bug reported by a dozen users. Real but low severity with a workaround; log it, reply to the reporters, and put it in the next sprint's queue.

What to communicate to whom

• Outage: status to the incident channel and affected customers' support contact, with next update time.
• Executive: a short message within the first hour with a firm commitment.
• Policy owner: a note that it is on track, no action needed.
• UI reporters: acknowledged, scheduled, not forgotten.

What slips and how to decide: the thing with the lowest cost of waiting. Here that is the UI bug, then policy (it has slack). Say what slips explicitly rather than silently dropping it.

Several asks at once: ask the requester which matters most and by when, a quick way to find real deadlines. Across a week the same logic applies: reserve time for incidents, scheduled jobs and deadlines, then fill the rest by importance.

What would change the order: if the outage grows or the executive's request is a regulatory filing, re-rank and tell people.

Direct answer

A bug that blocks half of users from the core flow is a severity-1 incident (the highest urgency level: a major outage or loss of a core function that justifies dropping planned work), so I stop and fix it now, and I make the sprint change explicit rather than letting committed stories slip silently. The order is: confirm and scope the bug, apply the fastest mitigation (an action that reduces user harm now, such as a rollback to the previous release or switching off the offending change), assign a small swarm (a few engineers pulled off other work to fix it together) to the fix, then decide which committed story leaves the sprint, and tell everyone who was counting on it.

Decision steps

1. Confirm (minutes). Reproduce it, check analytics for how many users are affected, and decide whether it truly blocks the core flow or only looks like it. Half of users unable to complete the core flow means the product is partly down.
2. Mitigate first. If the cause is a recent release, a rollback or a feature flag (a switch that turns a change off without redeploying) is usually faster than a fix. A reversible mitigation is preferred while the real cause is investigated.
3. Swarm. Assign one or two engineers with an incident lead (one person who coordinates the response and communication); keep the rest on committed stories so the whole team isn't pulled in.
4. Re-plan the sprint. The product owner (the person who sets priorities for the team's backlog) and team decide together what leaves. Choose the lowest-value story that is not started, rather than letting all three slip a little.

Worked example (illustrative)

Day 5 of a 10-day sprint, five engineers, so 25 engineer-days remain. Remaining work: story A 10 days, story B 8 days, story C 6 days, total 24, almost no slack. The swarm takes two engineers for two days, 4 engineer-days, leaving 21. A + B = 18, which fits with 3 spare days to absorb the bug tail. Dropping C (not started, lowest value) is the right call, announced on day 5 rather than discovered on day 10. C's new date is the end of next sprint, which from day 5 is about 15 working days away, so the message should give that date, not a vague "soon". If the fix drags on, the 3 spare days are the buffer (slack time kept free for surprises).

Communication

• Team: what changed, who is on the incident, what happens to story C.
• Stakeholders waiting on C: what moved, why, the new target date.
• Support and customers: a plain status message and any workaround, with the time of the next update.
• Leadership: impact, mitigation status, expected resolution, and the next update time, delivered on a fixed cadence until it is resolved.

After resolution, a blameless review (a post-incident discussion about process gaps, not individuals) looks at how the bug got through, not who.

Sample message to stakeholders waiting on story C: "Heads up: a bug is blocking about half of users from checkout, and two engineers are fixing it now. To protect the sprint, story C moves to next sprint, new target [date of the end of next sprint]. A and B stay committed, with about 3 days of buffer left. I will update you at 3pm today and when it is resolved."

The same method in other situations

• Small percentage but high-value customers: severity is not just a percentage. Weigh contract value, renewal risk and regulatory exposure against the fix cost. If there is a workaround, offer it and schedule the fix by value.
• Major incident versus a revenue feature: the incident normally wins, unless the feature has a hard contractual date with a larger penalty. Keep one person on the feature so it is not abandoned.
• Latency regression: a partial rollback or flagging off the offending change is a cheap, reversible way to buy time to fix properly.

Pitfalls

• Heroics instead of replanning. Absorbing the bug "somehow" while keeping all three stories ends in quiet failure.
• Whole-team swarm. It stalls everything else for little gain.
• Silent scope change. Surprises erode trust faster than bad news.