Knowledge Sharing and Team Enablement Questions

Direct answer

Start from one measurable objective ("by the end you can do X on your own"), make at least half the time hands-on (the worked example below is 100 of 180 minutes, about 56 percent, with the rest on a demo, share-back and admin), design labs with known expected outputs plus facilitator notes for likely mistakes, and gather feedback in three ways: during, at the end and after two weeks. Send pre-work before and follow-up materials after so people can build a first piece independently.

Worked example: a 3-hour workshop, "read and act on a service dashboard" (illustrative)

Objectives: by the end each participant can (1) find the latency and error-rate panels (individual charts on the dashboard) for a service, (2) tell whether a spike lines up with a deployment (a release of new code to the live service), and (3) write a two-line status note for it.

Pre-work (30 minutes): install nothing new; open the sandbox dashboard link and check login works. A one-page glossary defines p95 latency (the response time that 95 percent of requests beat) and error rate.

Lab 1: given a sample dashboard, find p95 latency for the checkout service in the last hour. Expected output: they report the value from the panel and the time window. Facilitator note: the most common mistake is reading average instead of p95; ask "what would a slow user see?"

Lab 2: a planted fault (a spike in errors at 14:05). Expected output: "error rate rose from about 1 to 8 percent at 14:05, matching the 14:03 deploy; suggest a rollback (going back to the previous version) and check logs." Facilitator note: if they blame the database, ask what evidence links it.

Evaluating participants' work

Use a rubric (a scoring guide) on the written status note, the short update a teammate who was not there would read:

Peer review: each person scores a neighbour's note against the rubric, then discusses. For a problem-solving and communication workshop, add rows for "states the next action" and "flags what is still unknown".

Feedback so the next run is better

• During: facilitator notes on where people got stuck and how long each lab really took.
• End: a 4-question form (one thing useful, one confusing, pace, confidence 1-5).
• Two weeks later: "Have you used this? What blocked you?", and follow-up materials (a checklist, the lab solutions, a starter template) for building a first piece alone.

Pitfalls

• Too much talking, too few labs.
• Labs with no expected output, so participants cannot tell if they are right.
• No timebox slack. If time runs short, shorten Lab 1 or limit the share-back to one presenter. Do not cut Lab 2: it is the only exercise that covers objectives 2 and 3 (linking a spike to a deployment and writing the status note), and cutting it would drop two of the three objectives.

Direct answer

I would prepare support around the decisions they will make, not the dashboard's features: for each alert, what it means, what to check first, and when to escalate. Then rehearse with realistic examples before go-live, and run a structured feedback loop in the first month, measuring whether their escalations are accurate.

Before launch (two to three weeks)

1. Alert cards. One short card per alert: meaning in plain words, severity, the first thing to check, who to escalate to and how, and an example. Link the card from the alert itself.
2. Training on real cases. A 60-minute session using past incidents replayed on the dashboard: "what does this alert mean, what do you do?" Include the alerts that look scary but are harmless.
3. Dry run. A week where support watches the dashboard alongside engineering with no obligations, then a few simulated alerts to test the escalation path.
4. One-page cheat sheet and a named engineering contact for the first weeks.

First month: collecting feedback

• A quick "was this alert clear?" thumbs up or down and comment on each alert card.
• A dedicated channel for questions, tagged by alert type.
• A 15-minute weekly review: top misread alerts, unclear cards, missing alerts, noisy alerts.
• Fix cards and alert wording the same week and tell support what changed.

What I would measure

• Escalation accuracy: escalations that engineering judged necessary divided by total escalations. Illustrative: 40 escalations, 28 necessary gives 28/40 = 70 percent. Track its trend by week.
• Time from alert to first support action.
• Number of "what does this mean" questions per week (should fall).
• Missed alerts: alerts that mattered but were not escalated (found in review).

Worked example: an alert card (illustrative)

Alert: Checkout error rate high
Means: More payment requests are failing than usual.
Check first: Is the payment provider's status page showing an outage?
Escalate if: still high after 10 minutes or provider status is normal -> page on-call payments engineer.
Ignore if: single spike under 2 minutes.

Pitfalls

• Training on every feature; support only needs actions per alert.
• Only measuring satisfaction. Use accuracy to see actual understanding.
• Noisy alerts teach people to ignore the dashboard. Feed noise reports back and tune alert thresholds.

Direct answer

A note on words: in this answer a page means an alert that notifies the on-call engineer (the person whose turn it is to respond to production problems, including out of hours), not a documentation page. A noisy alert fires often without needing action, and an escalation is handing a problem up to a more experienced person.

For the first 30 days I would listen and measure, not write. Then I would spend days 31 to 60 killing the top sources of firefighting with short runbooks and fixes, and days 61 to 90 building habits so that knowledge keeps spreading without me. The bet is that a small number of repeated problems cause most of the pain, so documentation aimed at them pays back fastest. A runbook is a short, step-by-step guide for handling a specific situation, such as a particular alert.

Days 1 to 30: learn and measure

• Read the last 30 to 60 days of pages and incidents. Tag each by service and cause.
• Interview each engineer: what do you dread being paged for, and what do only you know?
• Build a simple service map: what exists, who owns it, what depends on what.
• Ship one quick win: fix a noisy alert or write one runbook for the worst repeated page. It buys credibility.

Days 31 to 60: attack the top causes

• Write and test runbooks for the top 5 alerts. Have someone who did not write it follow it.
• Assign a named owner per service and a one-page service card (purpose, owner, dashboards, how to roll back).
• Pair on-call: the newcomer shadows, then leads with a backup.

Days 61 to 90: make it self-sustaining

• Every incident ends with a doc or alert change, tracked as a ticket.
• Add a short doc checklist to code review and a monthly review of the most-used pages.
• Set a target: fewer repeat pages, and fewer escalations to the one expert.

Worked example

The last month shows 60 pages. Grouping them, 5 alerts account for 39 pages: 39 / 60 = 65%. If runbooks and fixes for those five remove even half of those pages, that is 19 or 20 fewer pages a month, and the on-call load falls where it hurts most. I would report the actual figure at day 60 rather than promise it now.

How I would know it worked

Pages per week, repeat-page rate (pages from a cause seen before), share of pages resolved without escalating, and time from page to first correct action.

Trade-offs and pitfalls

• Do not try to document everything. Coverage is not the goal; fewer fires is.
• Do not become the single new expert. Success means others resolve the issue without you.
• Flip condition: if one service causes most pain and is truly unstable, stabilise it before writing docs about how to babysit it.

Direct answer

I conclude that the knowledge lived in one person's head rather than in the team: the restore path was undocumented or never practised, ownership of the system was ambiguous, and the on-call rotation (the schedule of who answers pages) did not require anyone else to be able to restore it. That is a system failure, not one engineer's failure, so the postmortem (the written review of an incident) should be blameless: it asks what allowed this, not who is at fault. Over the next quarter I would first stabilise, then make the restore repeatable by others, then prove it.

What the outage tells me

• Knowledge is tribal (held informally by a few people rather than written down): the restore steps exist only as habit.
• Ownership is unclear: nobody was named as owner of the system, and monitoring did not page the people who could act.
• The rotation tested availability, not capability: being on call did not mean being able to fix.
• The long recovery time (MTTR, mean time to recovery: the average time from an outage starting to service being restored) was driven by waiting for or reconstructing knowledge, not by the fault itself.

Immediate stabilisation versus durable fix

Quarter plan

1. Weeks 0 to 2: runbook written by pair (expert speaks, second engineer executes and fixes every step that does not work as written). The first draft must be run by someone who did not write it.
2. Weeks 2 to 4: assign one owner and one named secondary for the system in the service catalogue (the internal directory listing each system, its owner and how to reach them); route alerts to the rotation, not to the expert's phone, so ownership and monitoring line up.
3. Weeks 4 to 9: mentorship-led spreading: the secondary shadows every incident and change, then leads with the expert navigating; monthly brown-bags (short informal lunch talks) on how the system fails; add "can restore X" as an on-call readiness item before someone joins the rotation.
4. Weeks 9 to 13: a game day (a rehearsed failure, also called a drill; the two words mean the same here) where the expert is deliberately unavailable and someone else restores from the runbook.

Worked example (illustrative)

Suppose the billing database needed a point-in-time restore (rolling the database back to its exact state at a chosen moment) and only one engineer knew the sequence. The runbook lists eight steps. In the game day, engineer B executes it alone in staging and finds step 5 depends on a credential only the expert holds. That finding is the point: it costs an afternoon in staging rather than hours in production. I would count success as "two people other than the original expert can complete the restore from the runbook", plus the drill's recorded time. For example (illustrative): the first drill takes 95 minutes because of the credential gap, and the second, after the gap is fixed, takes 55 minutes with a different engineer, so the trend and the two independent successes are the evidence.

Trade-offs and pitfalls

• Documentation alone decays; the drill and the rotation change keep it true.
• Do not make the expert write everything in their spare time; that turns them into the bottleneck. Protect time and let the learner do the typing.
• Avoid a heroic blame narrative, because the expert will hide information next time.
• If the system is truly rare, the alternative is simplifying or automating the restore so it needs less knowledge.

Direct answer

I would run a separate, short retrospective across the run of incidents, aimed at one question: what did we not know, and why? Individual postmortems (the written review of one incident: what happened, why, and what to fix) ask what broke. This session looks across them for repeated knowledge gaps (a missing runbook, meaning a short step-by-step guide for a specific situation; an unclear owner; information held by one person) and ends with a small learning plan with owners and a metric. A retrospective here means a facilitated look back to learn, not to assign blame.

Preparation

• Pull 3 to 5 recent incidents and build a one-line timeline for each, marking the moments where the team was stuck or waiting for a person.
• Invite responders from different teams, not just managers. A responder is whoever was on call (the person whose turn it was to handle production alerts) or joined the incident.

Facilitation steps (about 60 to 90 minutes)

1. Set the tone: blameless (we examine the system and the information, not the person). State that at the start.
2. Silent writing: each person writes "the moment I wished I knew X" on notes, per incident.
3. Cluster the notes into themes.
4. Prompts for missing artifacts: was there a runbook, dashboard or diagram? Was it found, was it current?
5. Unclear ownership: who did we page first, and was that the right team? Who could have decided?
6. Tacit-only knowledge (knowledge that lived in one head and was never written): who did everyone ask, and what would happen if they were away?
7. Choose corrective actions (concrete follow-up tasks that close a gap): at most 3 to 5, each with an owner and a date.

Worked example

A sample note from silent writing: "Incident 2, 14:10: I wished I knew what the queue worker does when a job fails, and nobody but Sam knew."

Four incidents in a quarter. In three of them, responders lost time asking one engineer how the retry settings of a queue worked (how many times it re-attempts a failed job and how long it waits between tries). Ownership was unclear in two (nobody knew which team owned the shared cache, a common fast store several services read from). The plan: (1) write and test a queue runbook, (2) put a named owner in each service card (a one-page summary per service: purpose, owner, dashboards, how to roll back), (3) a 30-minute walkthrough recorded and linked from the alert, with a second person shadowing (watching and learning alongside) the queue expert for the next two on-call weeks.

Cross-team learning plan and success metrics

A plan row looks like: "Write and test queue runbook | owner: Priya | due: in 3 weeks | metric: runbook opened in the next queue incident".

Share the themes, not the names. Track: time to first correct diagnosis, how often responders escalate to the one expert, repeat incidents from a known gap, and whether the new runbook was opened during incidents.

Trade-offs and pitfalls

• Avoid a long list of actions. Few done beats many logged.
• Do not turn it into a blame session; people stop sharing.
• Flip condition: if the gaps are truly systemic (a missing platform), escalate as a planning request, not a documentation task.