Disaster Recovery and Business Continuity Questions

Direct answer

I treat a critical vendor as an extension of the organization's own criticality tiering, not as someone else's problem once a contract is signed. That means verifying the vendor's own continuity posture with evidence rather than a sales claim, classifying how much concentration risk it represents (how many of our functions have no alternative path if it fails), negotiating contractual continuity obligations up front, and deciding, before anything breaks, what the business needs to happen when the vendor is down. That last decision belongs to the business-process owner who depends on the vendor, not to whichever engineer is on call when the outage starts.

Structured elaboration

Verifying the vendor's own continuity posture. Ask for evidence, not assurance: the date and outcome of their most recent continuity or DR test, a relevant attestation (SOC 2 Type II, ISO 22301 or 27001 certification), their own sub-vendor dependencies (a vendor who is itself entirely dependent on one cloud region is a risk you've now inherited), and their actual incident history rather than their marketing page's uptime claim.

Classifying concentration risk, "single point of failure" in the supplier sense. This means a vendor whose outage stops multiple business functions or customers because there is no alternative path, the same idea as a technical single point of failure, just applied to a supplier relationship rather than a system component. Tier it the way a business impact analysis (BIA), the exercise that quantifies what an outage costs each business function and sets its recovery targets, tiers internal functions:

Contractual levers. Recovery commitments (the vendor's own stated recovery time objective (RTO, the target time to get a function working again) and recovery point objective (RPO, how much recent data the function can afford to lose, measured in time) as contract terms), a notification SLA for when the vendor itself has an incident, audit or right-to-test clauses, and exit or data-portability terms so switching away from the vendor is actually possible rather than theoretical. SLA credits are worth including but shouldn't be mistaken for protection: a financial credit compensates the contract, it doesn't recover a lost transaction or a missed regulatory deadline.

The business-level fallback decision. For each Tier 1 or Tier 2 vendor, the business-process owner decides, in advance, one of: accept the risk (documented, with a name and a date attached to that decision so it's revisited, not forgotten), qualify and periodically re-test a secondary vendor, stand up a documented manual or degraded-mode workaround with a stated capacity limit, or insource the function. What the technology needs to deliver follows from that decision; the decision itself is a business call informed by, not delegated to, engineering.

Worked example

A mid-size payments company depends on a single external identity-verification vendor to open new customer accounts. The concentration risk is total: if new-account signups drive roughly $80,000/day in first-year revenue and identity verification is the only path to opening an account, then every hour of vendor downtime puts a proportional share of that revenue at risk:

$$\$80{,}000 / 24 \approx \$3{,}333 \text{ per hour at risk}$$

Given that, the business-process owner (not engineering) sets the fallback: a manual review process run by the compliance team, using existing documentation to verify identity for a capped number of applications per hour, explicitly rated to absorb up to 4 hours of vendor downtime before the queue backs up faster than compliance can clear it. That 4-hour figure is the business's own RTO for this dependency, arrived at the same way a BIA sets any other recovery target, and it's what tells engineering what the fallback integration actually needs to support, not the other way around.

Trade-offs and pitfalls

SLA credits create a false sense of protection: a contractual credit worth a few hundred dollars does nothing to offset a day with $3,333/hour of exposure sitting behind it. Auditing every vendor to the same depth is unrealistic, which is exactly why the concentration-risk tiering has to gate how much due-diligence effort a given vendor gets. A "qualified backup vendor" that was tested once at onboarding and never revisited is a paper mitigation, not a real one, since vendor capabilities and your own dependency on them both drift over time. The trap specific to a technical audience: letting engineering quietly build a technical workaround to a second vendor without the contractual, compliance, and business-tolerance work behind it creates a shadow dependency that nobody has actually assessed or approved.

Direct answer

After a drill I run a structured post-exercise review that classifies every finding as people, process, or technology, scores it for severity, assigns a single named owner and a due date scaled to that severity, and requires an independent verification test before the finding can be closed, not just the owner's word that it's fixed. Individual findings then roll up into a small set of trend metrics that a recurring governance review looks at, so leadership can tell whether the continuity program is actually improving over time rather than just accumulating a backlog of open tickets.

Structured elaboration

Classification. People (training, staffing, or awareness gaps, like nobody being reachable at the right time), process (a missing, wrong, or unclear runbook step), or technology (a system or tooling failure). Many findings are genuinely two categories at once, for example an automated failover script that failed is a technology issue, but nobody catching the error before the drill is a process issue in the review or testing procedure itself; classify the root cause that, if fixed, prevents recurrence, not just the surface symptom.

Severity and ownership. Score each finding by business impact: does it threaten a documented recovery time objective (RTO, the target time to get a function working again) or recovery point objective (RPO, how much recent data the function can afford to lose, measured in time), or does it block declaration authority (who has the standing to formally declare and activate the plan) entirely? Assign one accountable owner per finding, with a sponsor who can escalate if it stalls. Critical findings get the shortest fuse and the most visible tracking.

Verification before closure. The distinction that matters most: "marked done" is not the same as "closed." A finding should only close once there's an independent verification test, ideally at the next scheduled exercise from the tabletop-to-full-scale ladder, confirming the fix actually works, not a self-report from the person who made the change.

Governance cadence and trend metrics. Individual findings feed a recurring review, for example a quarterly continuity steering meeting, that tracks two things at the aggregate level: how quickly findings are actually closing (with verification, not just ticket status), and whether the same root cause is recurring across drills, which indicates the underlying plan or system was never really fixed the first time. This is what turns a series of one-off reviews into evidence the program is maturing, and it's also the artifact regulators and auditors typically want (see the regulatory-obligations answer on this topic for what a compliance program expects to see documented).

Tooling. Track findings in a central, auditable system (a ticketing tool with status, owner, and due date, not a slide deck that gets archived and forgotten), since the whole point of the review is a defensible trail from finding to verified fix.

Worked example

A team ran four quarterly drills and tracked how many days it took to close each Critical finding, from the day it was logged to the day its verification test passed. In Q3, five Critical findings closed with these lead times in days: 12, 18, 21, 25, and 29.

$$\text{Average close time} = \frac{12+18+21+25+29}{5} = \frac{105}{5} = 21 \text{ days}$$

That 21-day average, tracked quarter over quarter alongside a second metric (the percentage of findings that recur in a later drill), is what the quarterly steering review actually looks at. A shrinking average close time with a low recurrence rate is real evidence the program is improving; a shrinking close time with a high recurrence rate usually means findings are being marked closed to hit the metric without the underlying gap actually being fixed, which is exactly why the verification-test requirement exists.

Trade-offs and pitfalls

Measuring only closure speed creates a perverse incentive to close findings before they're actually fixed, which is why speed has to be paired with a recurrence-rate metric, not tracked alone. Remediation work reliably stalls when the team is busy with feature delivery unless a named executive sponsor has the standing to protect capacity for it; without that sponsor, "we'll get to it" quietly becomes never. It's also worth being explicit about what this process is not: it's a review of the continuity plan and program itself, not a technical incident post-mortem of a system failure, so the findings and remediation often land on documentation, staffing, and ownership as often as they land on a system fix.

Direct answer. Design it as a functional exercise, a partial, real execution of the actual failover rather than just a tabletop discussion, run against a simulated outage of the dependency in a scoped, non-production environment. Success means the fallback route delivered what the business actually needs (transactions completed or safely queued, no duplicates, customers not silently left in the dark) within criteria set before the exercise runs. The rollback plan is a predefined abort trigger that reverts to the primary path and treats anything the fallback processed as needing reconciliation, exactly like a real incident would.

1. Scope and objectives

Name the specific dependency and specific fallback route under test (a secondary SMS provider, or a manual and queued path for payment confirmation), and state the objective in business terms: can the organization keep this function operating, at what level of degradation, for how long, if this specific dependency is unavailable. Choose the exercise type deliberately: a tabletop validates whether people understand the plan; a functional exercise, actually triggering the fallback and running real transactions through it in a controlled setting, validates whether it technically and operationally works. For a fallback route you're specifically trying to prove works under failure, a functional exercise is the right depth; a tabletop alone would only tell you the plan sounds right.

2. Design the scenario

Simulate the outage realistically but under control: route the primary dependency's calls to a fault response (timeouts, errors, or an explicit "unavailable" condition) in a scoped test environment with synthetic transactions and test accounts, never live customer data or real charges. Run more than one failure shape if the plan is meant to handle more than one: a hard outage behaves differently from a slow degradation (rising latency, intermittent errors), and can expose different gaps, particularly around whether the system correctly detects "bad enough to fail over" instead of tolerating it too long. Include volume, not a single transaction: a batch of concurrent test transactions through the fallback route surfaces queuing, ordering, and duplicate-handling behavior that one transaction at a time won't.

3. Define success before running it, not after

Business-facing criteria: every transaction attempted during the exercise either completes successfully via the fallback or ends up in a clearly tracked pending state, with zero duplicates and zero silently lost transactions, confirmed by reconciling the exercise's own transaction log against what the fallback provider and the primary system each show afterward. Operational criteria: the team followed the documented runbook to trigger the fallback rather than an ad hoc workaround, within whatever time target the plan specifies for detecting and switching over. Communication criteria, where the plan calls for customer-facing messaging during a real event of this kind: confirm the message that would have gone out is accurate to the actual state of the exercise's transactions, not a generic template nobody checked against what happened.

4. Rollback plan if the fallback itself misbehaves

Define the abort trigger ahead of time, not improvised mid-exercise: a concrete condition (error rate above a set threshold, a data-integrity check failing, an evidently wrong amount) that stops the exercise and reverts test traffic to the primary path, rather than letting a misbehaving fallback keep running and generating more to clean up. Treat anything the fallback processed before the abort exactly like a real incident: reconcile it, don't discard it, since finding out whether reconciliation actually works is as much the point of the exercise as testing the happy path. Have a named exercise controller with the authority to call the abort, separate from whoever is executing the steps, so the decision to stop isn't made by someone invested in proving the fallback works.

5. Close the loop

Run a hotwash immediately after (a short structured debrief on what worked, what didn't, and why) and log findings the way a real incident's post-event review would, with owners and target dates, so a gap the exercise found gets fixed and is retested at the next exercise rather than only known about.

Worked example

An organization tests its fallback SMS provider for one-time passcode delivery. Objective: if the primary provider is unavailable, can the fallback deliver codes within the plan's 30-second target with no duplicate sends. Design: a functional exercise, with primary-provider calls routed to a fault-injection endpoint returning timeouts for a fixed 20-minute window, run against 50 synthetic test accounts requesting codes continuously through that window, in a test environment with a provider sandbox that never sends real messages to real numbers. Success criteria set beforehand: at least 49 of the 50 attempts (allowing one edge case at the exact failover boundary) each receive exactly one code via the fallback within 30 seconds, and zero receive two.

Result: 50 of 50 delivered exactly once, averaging 22 seconds. But the exercise log shows the system took 40 seconds to detect the simulated outage and switch to the fallback in the first place, which isn't a delivery failure but eats into the 30-second budget for any request arriving in that detection window, and is logged as a finding. Nothing crossed the predefined abort trigger (an error rate above 5%), so the exercise ran to completion without needing rollback; the detection-time finding is written up with an owner and a target date for the next exercise to confirm it's fixed.

Trade-offs & pitfalls. Testing only the happy path, one transaction, no volume, is the most common shortcut, and it's exactly what misses the ordering and duplicate-handling problems that show up under concurrent load. Setting the success bar as "the fallback works at all" instead of "the fallback meets the actual business requirement" lets a technically-functioning fallback pass an exercise while still being unacceptable in a real event. And skipping the predefined abort trigger on the assumption an exercise "shouldn't need it" is how a test turns into a real incident; the rollback plan is itself part of what's being tested, not a formality around it.

Direct answer. Ownership has to be a named role tied to the business function or system the runbook covers, not "whoever wrote it last," with versioning that ties each revision to a specific trigger, not just a date. Catching drift before a real event means testing the runbook during a planned exercise instead of discovering it's wrong while executing it under pressure, paired with a lightweight review trigger whenever something the runbook depends on changes.

1. Ownership

Assign a named owner to every runbook, ideally the person or role accountable for the business function or system it covers, not a shared team inbox and not necessarily the original author, who may move on. Ownership means being responsible for the runbook's accuracy, not personally executing every step during a real event. Make it visible on the document itself (name, role, last-reviewed date) so anyone opening it during an incident can see who to ask if something looks wrong, and so a stale "owner" who's left the organization is an obvious red flag rather than a hidden one. When ownership changes hands, require an explicit handoff with a fresh review as part of the transition, not a silent reassignment.

2. Versioning

Treat runbooks as controlled documents: every change gets a version number, a date, who made it, and why, kept in the document's own change history rather than relying on file metadata or institutional memory. Tie versions to triggers, not just a calendar. A scheduled review is a reasonable baseline, but the more reliable trigger is linking review to the events that actually make a runbook wrong: the system it depends on changes, the org structure around it changes, or a real exercise or incident surfaces a gap. A runbook untouched for eighteen months right after the system it describes was rebuilt is far more suspect than one reviewed on schedule six months ago. Keep prior versions accessible, not just the latest, so a post-event review can tell whether a gap that showed up was already known and fixed but hadn't propagated to whoever was executing, or was genuinely new.

3. Catching drift before it matters

The strongest mechanism is using the runbook for real, on a schedule, inside an exercise (a tabletop discussion or a functional exercise that actually executes some steps), before a live event forces the discovery. A runbook nobody has walked through since it was written is a hypothesis, not a tested procedure. Build a lightweight checklist step into whatever process generates the kind of change that breaks runbooks: when a system change, a supplier change, or an org change is being planned, that process flags which runbooks might be affected, so the review happens close to the change instead of being discovered much later. Track a simple staleness signal across the whole runbook set (time since last review, time since the system or team it covers last changed) and report on it the way you'd report any other risk metric, so an accumulating backlog of unreviewed runbooks is visible to whoever owns the continuity program overall, not discovered runbook by runbook.

Worked example

A payments-processing runbook is owned by the payments platform lead, currently at version 4: v2 added a fallback provider, v3 corrected an escalation contact list after a reorg, v4 followed a functional exercise that found the documented rollback step no longer matched how the system actually rolled back. Six months after v4, the team migrates the payments platform to new infrastructure as part of an unrelated project. Because "infrastructure migration for a system with an owned runbook" sits on that project's change-review checklist, the payments platform lead is flagged automatically and confirms the runbook needs an update before the migration ships, catching the drift as part of the planned change rather than during the next incident. At the next scheduled functional exercise three months later, the team walks through the updated runbook end to end and finds one further step, a monitoring dashboard link, still pointing at the decommissioned infrastructure. That becomes a logged finding and v5.

Trade-offs & pitfalls. A purely calendar-based review cadence catches drift too slowly for a fast-changing system and too often for a stable one; tying review to real change triggers is more work to set up but matches effort to actual risk. Ownership without accountability for staleness (a name on a document nobody ever checks) is barely better than no ownership; the staleness signal has to surface to someone, not just exist. And testing a runbook only during a real event is the worst possible time to discover it's wrong: the exercise programme is what lets a team find the gap on an ordinary afternoon instead of during an actual outage.

Direct answer. Missing the recovery time objective (RTO, the maximum acceptable time to restore a function) by three times means there's at least one systemic gap, not just a rough edge, so the job is to find it with evidence rather than guesses, size exactly how big it is against the target, and hand leadership a small number of prioritized, owned actions they can fund and verify, not a wall of findings.

1. Reconstruct what happened, from evidence, not memory

Pull the exercise log: what step started when, what step finished when, who executed it, against the runbook's expected timeline. That produces a real segment-by-segment timeline to compare against the plan, not one end-to-end number. Interview the people who executed each step while it's fresh, and ask what they were unsure about or had to improvise, since a step that "worked" only because someone made an undocumented judgment call is itself a finding.

2. Quantify the gap in segments, not as one headline number

Break the recovery into the same phases the runbook uses (declaration and mobilization, technical recovery, validation and cutover) and compute actual versus target for each. A 3x miss overall is far more useful to leadership as "declaration ran 30 minutes over its target; everything else was close to plan" than as one undifferentiated multiple, because it points directly at where a fix has to go.

3. Get to root cause, not the first plausible explanation

Use a structured technique (repeatedly asking why a symptom occurred, tracing back through contributing causes) and validate each candidate against the actual exercise evidence before treating it as confirmed. Categorize causes, since each gets a different fix: a documentation gap (the runbook doesn't reflect how the system or org actually works now), a process gap (an approval or hand-off took longer than planned because the person with authority wasn't reachable), a skills gap (the team executing a step hadn't practiced it and had to work it out live), or a genuine technology gap. A drill that misses this badly usually has more than one of these stacked together.

4. Build a remediation plan leadership will actually fund

Prioritize by impact on closing the gap against effort and cost, and call out single points of failure separately since those carry outsized risk regardless of effort to fix. Give each item a named owner and a target date, not a team name; an action without a person's name on it is the most common reason remediation plans stall. Cap the executive-facing plan at a handful of headline items, each framed as a decision leadership needs to make (fund this, approve this change, accept this residual risk), and put the rest in a supporting appendix for the working team.

5. Present it so leadership acts, not just nods

Lead with the quantified gap and what it exposes the organization to in business terms, before the technical detail. Follow with the prioritized asks, then a committed date for the next validated drill so the fix gets verified rather than assumed.

Worked example

Target RTO for this function is 3 hours (180 minutes); the drill took 9 hours (540 minutes), exactly the reported 3x miss (540 = 3 × 180). Segment targets: declaration and mobilization 15 minutes, core technical recovery 75 minutes, validation and cutover 90 minutes (15 + 75 + 90 = 180, matching the overall target).

Segment actuals: declaration and mobilization took 45 minutes (30 minutes over target). Core technical recovery took 95 minutes (20 minutes over target) and tracked the runbook fairly closely. Validation and cutover took the remainder, 540 − 45 − 95 = 400 minutes, against its 90-minute target, a 310-minute overrun. Checking the totals: 30 + 20 + 310 = 360, and 540 − 180 = 360, so the three segment overruns account for the full gap. Validation and cutover alone is 310 of the 360 total overrun minutes, roughly 86% (310 ÷ 360 ≈ 0.86), and that's where the investigation focused.

The root cause traced back to a dependency the runbook never documented: a reporting service the order-management system silently required at startup, discovered only when the validation team couldn't get health checks to pass and spent hours troubleshooting before tracing it to the missing service. That's a documentation gap, not a technology gap, and it's exactly the kind of finding leadership can act on: fund a dependency-mapping and runbook-accuracy audit, with a named owner and a date, instead of a vague "improve documentation" line item.

Trade-offs & pitfalls. Presenting one blended "3x over" number invites leadership to ask for a single silver-bullet fix; segmenting the gap is what lets you ask for the specific investment that actually moves it. A remediation list with more than a handful of top-line items reads as noise to an executive audience and none of it gets prioritized. And the easiest wrong turn is treating the drill's outcome as an indictment of the people who executed it rather than the plan and documentation behind them; a blameless review is what actually gets people to admit where they improvised, which is where the real findings live.