Operational Risk Management Questions

Direct answer

Estimate the residual risk path by path. Inherent risk is the expected loss before any fix; residual risk is what remains after the fixes actually in place. Residual is inherent times one minus the verified effectiveness of those fixes, crediting only what has been tested, summed over paths. Put the result next to the tolerance (the most annual loss leadership says it can live with), turn the gap into costed roadmap items, and ask executives for the specific engineer-weeks that bring it under tolerance.

Estimate (illustrative, $k per year, expected loss = frequency x impact)

The failover credit is 30%, not 70%: an untested runbook is a control that exists on paper, so I credit only part of it. Report a range, not a point, and name the assumptions.

Feeding the roadmap

Suppose tolerance is $200k a year (set by leadership). Residual 380 is 180 over. Two items close it:

• Ship the canary rollout: 8 engineer-weeks, 80% effective, residual 0.6 x 0.2 x 250 = 30.
• Drill the failover until it works: 4 engineer-weeks, raising effectiveness to 70%, residual 0.5 x 0.3 x 400 = 60.

New total 90 + 60 + 30 = 180, under 200. (The gap of 180 and the new residual of 180 are a coincidence: one is how far over tolerance you were, the other is what remains.) That buys 380 - 180 = 200k a year of reduction for 12 engineer-weeks, about 16.7k a year per engineer-week.

Case to executives

Not "we should finish what we started" but: current residual 380 against tolerance 200, two named items, 12 engineer-weeks, resulting residual 180, with the revisit date and the consequence of waiting (the current exposure continues). Ask for a decision on those weeks.

Risks that stay above tolerance

If a risk is still above tolerance after the planned work, the risk register (the list of identified risks with owner, score and treatment) needs a treatment decision. Choose among: reduce further (add another control, such as the canary rollout), transfer (shift the loss to someone else, for example cyber or business-interruption insurance, or a contract clause making a vendor liable), avoid (stop the activity, for example retire the feature that causes the exposure) or accept (knowingly live with it). Acceptance needs a named executive, a written rationale and an expiry date; the item stays on the register with a KRI (key risk indicator: a metric that warns it is getting worse).

Pitfalls: crediting designed rather than proven controls, using one precise number from guessed inputs, and presenting residual without the tolerance it is compared to.

Direct answer. I would not choose between the two options: I would do both in sequence. This week I add immediate protections on my side (they need no vendor cooperation), measure the real impact, start a scoped fallback for the highest-value paths, and escalate to the vendor with evidence in parallel. I would not stake the service on an uncertain vendor timeline.

Decision process

1. Measure. Failure rate, which calls fail, revenue or user impact, and whether failures cluster in time. Example: 1,000,000 calls a month at 2% failure is 20,000 failed calls.
2. Classify the traffic. Reads that can be served stale (cacheable) versus writes that cannot (payments, orders). A cache only helps the first class.
3. Put a price on it. Compare the cost of the fallback with the loss it removes. Illustrative: if each failed call costs about $1 in lost margin and support, the 20,000 failed calls cost about $20,000 a month. A fallback that answers 70% of them saves about $14,000 a month; if it takes 6 engineer-weeks at about $4,000 a week ($24,000), it pays back in under 2 months. If the failing share were 0.2%, the loss would be $2,000 a month and the same build would take over 12 months to pay back, so I would only add timeouts and retries and press the vendor. Also weigh the vendor option as a probability, not a promise: a fix that might land in 3 months saves nothing in the meantime, and dedicated support does not remove failures that cluster in bursts.
4. Decide. If a large share of failures hit cacheable paths and impact is real, build the fallback now. If the impacted calls are non-cacheable, caching is the wrong tool: queue-and-retry or a second provider is.

Immediate mitigations (days, no weeks of build). Timeouts, retries with backoff, a circuit breaker and a cache or degraded mode do most of the work for most services; the later items matter only when the specific situation applies.

• Timeouts and a retry budget (a cap on how many retries you allow) with exponential backoff (each wait longer than the last, such as 1s, 2s, 4s) and jitter (randomised delay so clients do not retry in lockstep), honouring any Retry-After header (the vendor telling you how long to wait).
• A circuit breaker: after repeated failures, stop calling for a period and fail fast.
• Idempotency keys: a unique ID sent with each request so the vendor recognises a repeat of the same request and does the action only once, so a retry does not double-charge or double-create.
• Alerting on error rate, and a status-page subscription.
• Degrade gracefully: hide the feature or show cached data.

Illustration: suppose 70% of the failing calls are reads for which we hold a saved copy, possibly slightly stale. Those can be answered from the cache, so only the other 30% still fail for users: 20,000 x 0.3 = 6,000. One independent retry would cut a 2% failure rate to 0.04% (0.02 squared), but intermittent failures usually cluster in bursts, so retries help much less than that independence figure.

Long-term strategy. A thin adapter in front of the vendor so the code depends on an interface, not the vendor. Add a secondary provider on warm standby (already deployed and taking a trickle of traffic, so it can take over fast) for critical paths, and a cache with explicit staleness limits.

Vendor management

• Build an evidence packet (timestamps, request IDs, error codes) and use the named escalation contact, not the general support queue.
• Read the SLA: a 99.9% monthly SLA allows 43.2 minutes of downtime per 30 days (43,200 minutes x 0.001), and credits rarely cover lost revenue. Our own availability is bounded by the vendor's, since both must work.
• At renewal ask for notice periods on limit changes, termination for chronic failure, data export rights, and a better SLA.

Rate-limit change with no notice. Same toolkit: client-side throttling (a token bucket: tokens refill at a fixed rate and each request spends one, so short bursts are allowed up to the bucket size while the long-run rate stays capped), backoff on 429 responses (the HTTP status meaning 'too many requests'), request coalescing (merging identical simultaneous requests into one call to the vendor), shedding low-priority calls first (dropping the least important work to protect the important), then a cross-functional response: product reduces calls, finance prices a higher tier, procurement renegotiates.

Single-source authentication or search vendor. Short term: cache sessions or tokens so signed-in users continue, keep a break-glass path (an emergency, audited bypass that lets staff in when the vendor is down). Long term: abstraction layer, a second vendor tested periodically, and contract changes (exit rights, SLA credits, change notice).

Pitfall. Building for weeks without data on what is failing.

Direct answer. The register died because it was a document nobody needed, not because people are lazy. I would rebuild it as a small, owned, time-boxed set of entries that live in the same tracker the teams already use, with a fixed review rhythm and automatic nudges, and I would start by deleting most of what is there.

Step 1: reset the content (week 1 to 2). Freeze the old register and triage every entry. Illustrative: of 120 entries, merge 30 duplicates (leaves 90), close 20 that no longer apply (leaves 70). For the 70, each gets an owner who accepts it by name, or it is closed. An entry with no owner is not a risk, it is a note. Re-score the remaining entries on a single scale so scores are comparable.

Step 2: a minimal entry. Scenario in one sentence (cause, event, consequence), owner, score (likelihood and impact), a KRI (key risk indicator, a measurable early-warning signal), treatment (avoid, reduce, transfer or accept), next action with date, review-by date. Anything more is optional because every extra field is a reason to skip updating. The four treatments mean: avoid (stop the activity that creates the risk), reduce (add controls or work that lower likelihood or impact), transfer (shift some of the loss to someone else, for example through insurance or a contract clause) and accept (knowingly live with it, with a named owner and an expiry date).

A filled-in entry (illustrative):

ID:           RISK-014
Scenario:     Only one engineer knows the month-end billing job (cause); if they are out or leave (event), invoices go out late (consequence).
Owner:        Director of Platform Engineering
Score:        likelihood 3 x impact 4 = 12
KRI:          days since a second engineer last ran month-end unaided (red if over 90)
Treatment:    reduce
Next action:  second engineer shadows, then runs, month-end (linked ticket, due in 45 days)
Review-by:    90 days from today

The KRI is the early-warning number that tells you the exposure is growing before the failure shows up.

Step 3: connect it to the work tracker. Each risk is an issue in the tracker (Jira, for example) with its own type and states: Identified, Assessed, Treating, Monitoring, Accepted (with an expiry date), Closed. The treatment actions are ordinary linked tickets in the owning team's backlog, so risk reduction competes for capacity visibly, instead of living in a parallel document. The wiki page (Confluence, for example) becomes a generated view, not the source of truth. Notification rules: a nudge to the owner 7 days before the review-by date; an escalation to the owner's manager 14 days after it; an accepted risk reopens automatically on its expiry date. Entries tied to a vendor or customer contract carry the contract ID from the contract system or CRM (customer relationship management tool) and the renewal date, so contract risks resurface before renewal.

Step 4: the routine. Weekly: team risk item in an existing meeting (10 minutes, only changes). Monthly: cross-team review of the top risks and any overdue items. Quarterly: leadership review and an expiry audit of accepted risks (check that every accepted risk was re-decided on or before its expiry date, not left to run on).

Step 5: keep it fed. Initial and ongoing sources: incident postmortems (every postmortem must either add or update an entry), top paging alerts, architecture review findings, audit findings, the vendor list, and single-point-of-failure reviews (looking for any component or person whose loss alone would stop a service). Automation flags staleness: no edit in 90 days, missing owner after reorg, or a linked action overdue.

Measures that show it is alive. Share of entries with a live owner (target 100%), median age since last review, share of actions closed on time.

Pitfalls. Rebuilding the same spreadsheet; making the register a compliance chore with 20 mandatory fields; treating an empty register as success; and letting acceptance be permanent.

Direct answer

A recurring minor incident becomes a systemic operational risk when the cause sits in something shared (a library, pipeline, process or practice) rather than in any one team's work, so fixing each instance locally never stops the next one. I would watch for the signals below, write it up as a risk with evidence and a cost, take it to the owner of the shared thing, then to a leadership forum with a specific funding or ownership decision to make.

Signals that one-off has become systemic

• Same contributing factor, different teams. Tag postmortems (written reviews of an incident) by contributing factor (a condition that helped cause it, such as "config pushed without validation"), not by service, so a pattern across owners is visible.
• Not decaying. Count per week stays flat or rises after local fixes.
• Fixes that do not hold. The same failure returns after a "fix".
• Severity drift. Minor incidents or near misses start producing sev2 events (sev2 means severity 2, a serious incident; sev1 is the worst, and levels vary by company).
• Cost adds up. Cumulative engineer time exceeds the cost of a root fix.
• Customers or support notice first, so detection is also failing.

A working rule (illustrative, tune it): the same factor in 3 or more incidents across 2 or more teams within 8 weeks opens a systemic-risk review. Why those numbers: two incidents can be coincidence, three begins a pattern; two teams shows the cause is not one team's habit; eight weeks keeps the evidence recent.

Worked example

9 incidents in 8 weeks, 4 teams, all "config pushed without validation." Each took about 3 hours with 4 engineers involved, so 9 x 3 x 4 = 108 engineer-hours, or 108 / 8 = 13.5 hours a week (about 702 a year, 108 x 52 / 8). A shared validation step in the deploy pipeline is estimated at 80 engineer-hours, so it pays back in about 80 / 13.5 = 5.9 weeks, before counting customer impact, and assuming the shared step removes most of the recurrence.

Escalation and getting it fixed

1. A one-page risk entry (one row of the risk register, the running list of known risks with owners and scores): statement, linked incidents, cost, proposed treatments, and a key risk indicator (KRI, a measure that warns before harm), such as the share of config pushes that skip validation, with incidents per month carrying that factor as the lagging confirmation.
2. Walk the owner of the shared component through it first, so no one is surprised.
3. Bring a specific decision to the engineering leadership review: who is the single accountable owner, and what capacity is funded.
4. Track it in the register until the KRI drops, then close.

Pitfalls

Judging by count alone (check the total incident trend too); blame framing (writing it up as a person's mistake instead of a system gap); closing the risk when the ticket closes instead of when the signal drops.

Direct answer. Expected annual loss (EAL) is probability per year x loss per occurrence. R1 = 10,000, R2 = 2,000, R3 = 500, total 12,500. R1 ranks first. But the ranking alone should not decide the order of mitigation, and the estimate I trust least is R1's 2%.

Calculations

R1: 0.02 x 500,000 = 10,000
R2: 0.10 x  20,000 =  2,000
R3: 0.50 x   1,000 =    500
Total               = 12,500

R1 is 80% of the total (10,000 / 12,500), R2 16%, R3 4%. Ranking: R1, R2, R3. Intuition: an average, and the rare large loss dominates it.

Why the ranking is not the decision

1. Mitigation cost and effect. EAL is what you could save, not what it costs. If removing R3 costs $200 and R1 costs $400,000, R3 may come first. Rank by loss removed per dollar.
2. Shape of the loss. R1 is a once-in-fifty-years style event that arrives as a $500k hit; the chance of at least one in ten years is 1 - 0.98^10, about 18%. If the company cannot absorb $500k, that is a survival question an average does not show. R3 is a small, frequent cost (even odds in any given year): a cost of doing business to automate or accept.
3. Non-financial impact. Safety, legal or reputational harm does not fit the dollar figure.
4. Correlation. If R1 and R2 share a cause, fixing it removes both.

Which estimate I trust least: R1's probability. A 2% rate is rarely backed by history, so it is a judgement. The sensitivity is large: at 1% R1 is 5,000, at 4% it is 20,000. R3 is the most trustworthy because a 50% annual event happens often enough to count. R2's $20,000 loss may be biased too, but its probability has data behind it. Break-even check: R1 stays above R2 unless its probability falls below 0.4% (2,000 / 500,000), so R1's rank is fairly stable, but its size is not.

Spreadsheet sensitivity. Put the three rows in a sheet with low/likely/high for probability and loss, and recompute the ranking. If the order holds across the range, act on it; if it flips, estimate more carefully only where it flips.

Pitfall. Presenting 12,500 as a precise forecast. It is a planning figure.