Knowledge Sharing and Team Enablement Questions

Direct answer

I would spread the skill through real work, not extra meetings: a rotating "debug driver" role in which a learner handles live problems while the expert coaches from the side, backed by recorded debugging sessions that the other time zones can watch asynchronously. I would cap the expert's teaching time at a stated budget so delivery barely moves, and I would prove the transfer worked with a drill the expert does not help with.

Structured elaboration

1. Rotation and pairing (learn by doing).

• Each learner takes a two-week stint as secondary on-call (on-call is the rota of people who get alerted, or "paged", when the live system breaks; the secondary shadows the primary, the one paged first) for the tricky system, then a stint as primary with the expert as backup.
• Use reverse pairing: the learner drives (types, chooses the next probe) and the expert navigates by asking questions ("what would you check next, and why?"). Answers, not commands.
• Rotate across time zones so each region ends up with at least two people who can debug the system, otherwise the skill is still a single point of failure at 3 a.m.

2. Make it asynchronous-friendly.

• Record the expert's real debugging sessions with narration (screen plus voice) and attach them to the ticket. Add a two-line "how we found it" note: the symptom, the first wrong theory, what disproved it, the actual cause.
• A weekly 30-minute debug clinic run live in the overlap hours and recorded for others; one recurring bug is dissected each week.

3. Ownership model. Every critical subsystem names a primary and a secondary owner (in a CODEOWNERS file, the repository file mapping paths to responsible reviewers, or an ownership page). The secondary is never the expert.

4. Validate the transfer. A cold drill: replay a past incident or inject a known fault in staging (a safe copy of the production system where mistakes hurt nobody). The learner must find the root cause (the underlying reason, not just the visible symptom) within a time box (a fixed limit, for example 45 minutes); the expert may only answer yes or no. Pass means the expert's absence would not have changed the outcome.

5. Stop redundancy decaying. Keep rotating (skills fade after a few months without use), repeat a drill every quarter with a different learner, and track that each subsystem still has two people who passed.

6. Documentation and velocity. Documentation is a 15-minute by-product of each debugging session, not a separate project. Protect delivery with a budget, for example the expert gives about four hours a week, and the pairing happens on tickets the team was going to do anyway. Expect a small slowdown in the first weeks and say so up front.

Worked example

A team of six spans three regions: Berlin (Priya, Sam), Bangalore (Ravi, Meera) and US West (Dana, Omar). Priya is the only person who can debug the payments queue, so today only one region has anyone.

• Weeks 1-2: Sam (Berlin) is secondary on-call while Priya leads and handles the pages; two sessions are recorded.
• Weeks 3-4: Sam drives, Priya navigates. Cold drill for Sam. Berlin now has two people (Priya, Sam).
• Weeks 5-8: Ravi (Bangalore) watches Sam's recordings first, then pairs with Sam in the overlap hours; then Meera learns from Ravi, with Ravi navigating. Cold drills for both. Bangalore now has two.
• Weeks 9-12: the same cycle for Dana and Omar (US West), starting from the recordings. Cold drill for each.

Priya moves to backup for this system once all five learners have passed. Later cycles are taught mostly by earlier learners, so Priya's coaching time drops after the first cycle.

Trade-offs and pitfalls

• Pure documentation does not transfer debugging judgement; pure shadowing without driving leaves the learner passive.
• If the expert is about to leave, compress this into a recording-first, short-notice plan instead.
• If the team is small enough that pulling the expert for coaching would break a deadline, buy time by narrowing to the two or three highest-risk failure modes first.

Direct answer

I conclude that the knowledge lived in one person's head rather than in the team: the restore path was undocumented or never practised, ownership of the system was ambiguous, and the on-call rotation (the schedule of who answers pages) did not require anyone else to be able to restore it. That is a system failure, not one engineer's failure, so the postmortem (the written review of an incident) should be blameless: it asks what allowed this, not who is at fault. Over the next quarter I would first stabilise, then make the restore repeatable by others, then prove it.

What the outage tells me

• Knowledge is tribal (held informally by a few people rather than written down): the restore steps exist only as habit.
• Ownership is unclear: nobody was named as owner of the system, and monitoring did not page the people who could act.
• The rotation tested availability, not capability: being on call did not mean being able to fix.
• The long recovery time (MTTR, mean time to recovery: the average time from an outage starting to service being restored) was driven by waiting for or reconstructing knowledge, not by the fault itself.

Immediate stabilisation versus durable fix

Quarter plan

1. Weeks 0 to 2: runbook written by pair (expert speaks, second engineer executes and fixes every step that does not work as written). The first draft must be run by someone who did not write it.
2. Weeks 2 to 4: assign one owner and one named secondary for the system in the service catalogue (the internal directory listing each system, its owner and how to reach them); route alerts to the rotation, not to the expert's phone, so ownership and monitoring line up.
3. Weeks 4 to 9: mentorship-led spreading: the secondary shadows every incident and change, then leads with the expert navigating; monthly brown-bags (short informal lunch talks) on how the system fails; add "can restore X" as an on-call readiness item before someone joins the rotation.
4. Weeks 9 to 13: a game day (a rehearsed failure, also called a drill; the two words mean the same here) where the expert is deliberately unavailable and someone else restores from the runbook.

Worked example (illustrative)

Suppose the billing database needed a point-in-time restore (rolling the database back to its exact state at a chosen moment) and only one engineer knew the sequence. The runbook lists eight steps. In the game day, engineer B executes it alone in staging and finds step 5 depends on a credential only the expert holds. That finding is the point: it costs an afternoon in staging rather than hours in production. I would count success as "two people other than the original expert can complete the restore from the runbook", plus the drill's recorded time. For example (illustrative): the first drill takes 95 minutes because of the credential gap, and the second, after the gap is fixed, takes 55 minutes with a different engineer, so the trend and the two independent successes are the evidence.

Trade-offs and pitfalls

• Documentation alone decays; the drill and the rotation change keep it true.
• Do not make the expert write everything in their spare time; that turns them into the bottleneck. Protect time and let the learner do the typing.
• Avoid a heroic blame narrative, because the expert will hide information next time.
• If the system is truly rare, the alternative is simplifying or automating the restore so it needs less knowledge.