Operational Risk Management Questions

Direct answer

Estimate the residual risk path by path. Inherent risk is the expected loss before any fix; residual risk is what remains after the fixes actually in place. Residual is inherent times one minus the verified effectiveness of those fixes, crediting only what has been tested, summed over paths. Put the result next to the tolerance (the most annual loss leadership says it can live with), turn the gap into costed roadmap items, and ask executives for the specific engineer-weeks that bring it under tolerance.

Estimate (illustrative, $k per year, expected loss = frequency x impact)

The failover credit is 30%, not 70%: an untested runbook is a control that exists on paper, so I credit only part of it. Report a range, not a point, and name the assumptions.

Feeding the roadmap

Suppose tolerance is $200k a year (set by leadership). Residual 380 is 180 over. Two items close it:

• Ship the canary rollout: 8 engineer-weeks, 80% effective, residual 0.6 x 0.2 x 250 = 30.
• Drill the failover until it works: 4 engineer-weeks, raising effectiveness to 70%, residual 0.5 x 0.3 x 400 = 60.

New total 90 + 60 + 30 = 180, under 200. (The gap of 180 and the new residual of 180 are a coincidence: one is how far over tolerance you were, the other is what remains.) That buys 380 - 180 = 200k a year of reduction for 12 engineer-weeks, about 16.7k a year per engineer-week.

Case to executives

Not "we should finish what we started" but: current residual 380 against tolerance 200, two named items, 12 engineer-weeks, resulting residual 180, with the revisit date and the consequence of waiting (the current exposure continues). Ask for a decision on those weeks.

Risks that stay above tolerance

If a risk is still above tolerance after the planned work, the risk register (the list of identified risks with owner, score and treatment) needs a treatment decision. Choose among: reduce further (add another control, such as the canary rollout), transfer (shift the loss to someone else, for example cyber or business-interruption insurance, or a contract clause making a vendor liable), avoid (stop the activity, for example retire the feature that causes the exposure) or accept (knowingly live with it). Acceptance needs a named executive, a written rationale and an expiry date; the item stays on the register with a KRI (key risk indicator: a metric that warns it is getting worse).

Pitfalls: crediting designed rather than proven controls, using one precise number from guessed inputs, and presenting residual without the tolerance it is compared to.

Direct answer

Compare the two paths in money and in tail risk, not in gut feel. Accept the risk only if (1) the expected cost of accepting plus the contingency is lower than the cost of the delay, (2) the worst case is survivable and reversible, and (3) someone with the authority to own that loss signs for it. In the numbers below accepting is cheaper over the first year, but only until the probability estimate moves or the risk stays open for a second year, so the estimate and the contingency's proven readiness are what I would pressure-test.

Worked example (all figures illustrative)

Expected loss means probability times impact, per year. Tail risk means a rare but very severe outcome, the kind an average hides. The three lines model splits risk work into a first line (the teams that build and run the product and own its risks), a second line (security, risk and compliance specialists who advise and challenge the first line) and a third line (internal audit).

• Impact if the risk occurs: $2,000k. Chance in the first year: 20%. Expected loss = 0.20 x 2,000 = $400k.
• Contingency plan (tested rollback, kill switch, customer communication) cuts the impact by 60%, so residual expected loss = 400 x 0.40 = $160k, plus $40k to build and maintain the plan: $200k to accept.
• Preventive control: removes the risk, but the six-week delay costs about $300k in deferred revenue and team time: $300k to eliminate. The $300k delay is paid once, while the $160k expected residual loss recurs each year the risk stays open. So $200k against $300k is a first-year view: over two years accepting costs 40 + 2 x 160 = $360k, more than $300k, unless a later fix closes the risk (which is why the acceptance gets an expiry date).
• Break-even probability p: p x 2,000 x 0.40 + 40 = 300, so p = 32.5%. At 20% accepting wins by $100k. At 40% accepting costs 0.40 x 2,000 x 0.40 + 40 = $360k and delaying wins.
• The break-even depends on how long the risk stays open, because the delay is paid once and the residual loss recurs. Over two years: p x 800 x 2 + 40 = 300, so p = 260 / 1,600 = 16.25%. Turned around, at the 20% estimate (assumed to repeat each year) accepting costs 40 + 160n for n years open, which equals $300k at n = 260 / 160 = 1.625 years. So the 20% estimate favours accepting only if the risk is closed (by a later fix) within about a year and a half, which is exactly what the expiry date and the funded follow-up enforce.

Who has to agree

• The risk owner: the executive who owns the business outcome (product or general manager), because they absorb the loss. Engineering recommends but does not accept on its own.
• Security or risk (second line, the specialists who challenge the first line's assessment) confirms the assessment is honest and the residual sits inside risk appetite (how much risk the company is willing to pursue) and the delegated authority of the signer. Above that limit it escalates.
• Legal or compliance if a contract or regulation is involved. A signature cannot waive a legal obligation; counsel decides that.
• Write it down: risk, options, numbers, signer, expiry date, review trigger.

What would change my answer

• The probability estimate and the time the risk stays open (the 32.5% first-year break-even is the line to watch, and it falls to 16.25% if the risk stays open two years).
• Whether the contingency has actually been exercised. A plan that exists on paper has only been designed (design effectiveness: it should work if followed); one that has been drilled has shown it works in practice (operating effectiveness). Count the 60% reduction only for the drilled kind.
• The tail: if the loss is irreversible (data loss, safety, regulatory breach), a contingency does not make it acceptable.
• A middle path often beats both: ship behind a feature flag to a small share of users, cut scope, or implement a smaller preventive control in two weeks.
• A hard external deadline raises the cost of delay and shifts the answer toward accept.

Pitfalls: letting the loudest stakeholder set the probability, treating "strong contingency" as proven, and accepting with no expiry so the exception becomes permanent.

Direct answer

Technical debt means shortcuts in design or code that save time now and cost more to maintain or fix later. Treat it as a decision that belongs to the person who owns the outcome, and make that decision informed, explicit and written down. I do not try to win the argument or block the milestone. I make the risk concrete in the stakeholder's own terms, offer options that protect the milestone, and if they still accept, I record who accepted what, until when, and what triggers the fix.

Negotiation, in order

1. Understand the milestone. What does hitting it buy (a contract, a launch, funding)? If I cannot say what they gain, I will not be heard on what they risk.
2. Quantify the risk in their language. Example (illustrative): a 25% chance in the next two quarters that the debt causes an outage costing $800k is an expected loss of 0.25 x 800 = $200k, plus the customer trust cost I cannot price. Show a range rather than false precision.
3. Offer options, not a veto: a reduced-scope mitigation that fits before the milestone, a time-boxed acceptance with a funded remediation slot next quarter, or a tripwire (a key risk indicator, KRI: a metric that warns the risk is getting worse) that automatically reopens the decision.
4. Ask for an explicit decision from someone with the authority to accept it, not a vague nod in a meeting.

Documenting the decision (risk acceptance record)

Accountability if it manifests

The signer owns the business consequence and engineering owns executing the contingency. The incident review uses the record as the starting fact pattern (what was known, what was accepted, were the conditions kept), blameless in tone (it asks what in the system and process allowed the outcome, not whom to punish) but not blind to who decided. Remediation was pre-scheduled, so it starts immediately rather than being renegotiated.

Escalating to leadership and legal

If the stakeholder lacks authority for this size of risk, or the debt touches a legal duty (data protection, contractual uptime commitments), escalate. If an interviewer asks for a real example, tell it in STAR form (Situation, Task, Action, Result, a standard format for behavioural answers). Situation: state the architectural risk and the deadline pressure. Task: get a decision from the right authority. Action: send a written one-page summary with evidence (incident history, load data, assessment), the options, and the decision requested by a date, copied to legal where an obligation is involved. Result: report what was decided and what changed. If you have no direct experience, say so honestly and describe a smaller comparable case (for example, a time you documented a disagreement over a release) plus exactly the steps above you would take. Do not invent an outcome.

Pitfalls: becoming the "no" person, accepting verbally with nothing in writing, and acceptance without an expiry.

Direct answer. I would not treat this as ship or do not ship. I would price all three options, and unless the numbers say otherwise I would limit the rollout: staged release behind a feature flag with automatic pull-back. Staging keeps most of the upside while cutting the 40% regression's blast radius (how many users and how much revenue a failure can touch).

1. Put numbers on it (illustrative: the feature surface earns $1M per month, so a 30% lift is $300,000 per month, which is $300,000 x 12 / 52 = about $69,231 per week)

• Expected loss of full release = 0.40 x $500,000 = $200,000.
• Gain from the lift over 3 months is $900,000, so full release has an expected net of about $700,000. Shipping is clearly favoured over not shipping; the question is how.

2. Staged rollout versus full release. Both options are priced on one basis: dollars of lift given up plus expected regression loss. Assume weeks 1 and 2 run at 10% and then 50% of traffic (these are the ramp steps, the traffic shares the feature moves through), and that guardrails catch 80% of regressions while only the canary (the first, small slice of traffic) is exposed. Full release gives up no lift, so its cost is just the $200,000 expected loss.

• Foregone lift (the lift we do not collect while most users are held back): week 1 loses 90% of $69,231 = $62,308; week 2 loses 50% = $34,615; total about $96,923.
• Loss if the regression is caught at 10% exposure: only 10% of users are hit, so 10% x $500,000 = $50,000.
• Expected loss: the regression occurs 40% of the time; of those, 80% are caught at 10% exposure ($50,000) and 20% reach everyone ($500,000). So 0.4 x (0.8 x 50,000 + 0.2 x 500,000) = 0.4 x 140,000 = $56,000.
• Total expected cost of staging = $96,923 + $56,000 = about $152,923, versus $200,000 for full release, a saving of about $47,000. The 80% catch rate is an assumption; at a much lower catch rate the advantage shrinks, which is why detection quality matters.

3. Delay is justified only if the lift is smaller than the expected loss, or the loss exceeds what the business can absorb, or a fix would cut the 40% cheaply. Otherwise delay forfeits value (a week costs about $69,000 of lift).

4. Guardrails

• Pre-agreed pull-back thresholds on error rate, latency and a business metric (conversion), tied to the SLO error budget (the failures an SLO allows).
• Feature flag kill switch, tested before launch, with automatic rollback for fast breaches.
• Named owner on call, ramp steps that only advance when metrics hold for a fixed window.

Trade-offs. Expected value ignores that a $500k outage may also cost trust. If the downtime could breach a contract or hurt safety, I weight the downside more.

Direct answer

Score each action by the risk it removes per engineer-week (one engineer working for one week), evaluated in sequence because later fixes act on what earlier ones left. Do the cheap, high-ratio mitigations now with an expiry date, commit the redesign only if the remaining risk is above tolerance or the redesign removes a whole class of failures, and communicate it to leadership as dollars and weeks.

Framework

1. Baseline the failure mode's expected loss per year. Illustrative: $500k.
2. Estimate each action's effort and what fraction of the remaining risk it removes. Replay the last few incidents and ask of each: would this action have prevented it, or cut its size, and by roughly how much? The percentages in the table are illustrative outputs of that exercise, not measurements.
3. Order by ratio; apply in sequence (each reduces what is left, not the original).
4. Compare the remaining risk to the stated tolerance (the most annual loss leadership says it can live with for this failure mode).
5. Attach a revisit date to every patch so a stopgap does not become the plan.

Worked example ($k per year, illustrative)

The first three (10 engineer-weeks) remove about 293 of the 500, roughly 59%. The last two cost 150 engineer-weeks for about 182 more.

Decision against tolerance

If leadership's tolerance for this failure mode is $150k a year, the retry rewrite (124 remaining) gets under it, and the cell architecture does not pass on this evidence alone. It would need another justification: it removes a whole class of failures the patches do not touch, or it frees the toil (repetitive manual operations work) of maintaining patches.

Communicating the trade-offs

Offer three packages in one table: patches only (10 weeks, 207 remaining), patches plus retry rewrite (40 weeks, 124 remaining), everything (160 weeks, 25 remaining). Say what is uncertain (the percentages are judgement) and what each package does for incident customers feel. Let leadership choose the tolerance; do not choose it for them.

Pitfalls: summing gains as if independent (it overstates), letting stopgaps stay forever, and defending the redesign with ratio alone when its value is structural.