Knowledge Sharing and Team Enablement Questions

Direct answer

I would not answer "did it work?" with one number. I would run five analyses: who took part compared with who did not, change in assessed skill, change in real behaviour against a comparison group, and, separately, quality of interpretation (do people read analytics correctly) as opposed to tool usage, plus a cut of all of these by function. Then I would give leadership a segmented verdict: what the evidence supports, what it does not, and what I would do next, including stopping parts of the programme.

The analyses

(A cohort is a group of people who start the programme at the same time. Selection means the people who chose to take part differ from those who did not, so their results may reflect who they are, not what the programme did.)

1. Participation and selection: who enrolled, who finished, by function and seniority. If only already-keen analysts finished, any gain is partly the people.
2. Pre/post assessment by cohort: the same scenario test at start, end and three months later, compared cohort by cohort.
3. Usage behaviour with a comparison group: weekly dashboard use for participants versus similar non-participants (people I pick to match participants on function, seniority and baseline dashboard use, so the two groups started in the same place), before and after. This is difference-in-differences: the change in the trained group minus the change in the untrained group, which removes company-wide trends.
4. Interpretation quality: a scenario test where the answer is right or wrong (a truncated axis, meaning a chart whose vertical axis starts above zero so small differences look large; correlation read as cause; a percentage without its base number) plus a rubric review (scoring each document against a fixed checklist of what correct use of numbers looks like) of a sample of 30 recent decision documents for correct use of numbers. Usage counts show a tool was opened, not that a chart was understood.
5. Segment cut: results by function, since "mixed" often means strong in some teams and flat in others.

Worked example (illustrative numbers)

• Participation: 200 people were eligible, 60 enrolled and 48 finished. Only 50 of the 200 were analysts, yet 30 of the 48 finishers were analysts: 30/50 = 60% of analysts finished versus 18/150 = 12% of everyone else. So the finishers are mostly the people who already used data, and any gain must be read with that in mind.
• Pre/post by cohort: cohort A (24 people) scored 4.0 at start, 4.8 at the end and 4.4 three months later. Cohort B (24 people) scored 4.0 then 4.4 at the end, and its three-month re-test is not due yet. The average end score is (4.8 + 4.4) / 2 = 4.6, which is the participant figure used below. Cohort A already lost half of its gain, a sign of recall rather than habit.
• Segment cut, usage gain per participant: analysts (30 people) +2.0 sessions a week, operations (10) +1.0, support (8) +0.25. Weighted: (30 x 2.0 + 10 x 1.0 + 8 x 0.25) / 48 = 72 / 48 = +1.5, matching the overall participant gain below. The average hides that support barely moved.
• Usage: participants averaged 1.0 dashboard sessions a week before and 2.5 after (+1.5). Comparable non-participants went from 1.0 to 1.5 (+0.5). Effect on usage: 1.5 - 0.5 = +1.0 session a week.
• Interpretation score out of 8: participants 4.0 to 4.6 (+0.6), non-participants 4.0 to 4.4 (+0.4). Effect: 0.6 - 0.4 = +0.2 of a point.
• Is +0.2 real or noise? It is 2.5% of an 8-point scale. If individual scores commonly differ from each other by a point or more (an assumption I would check in the data) and each group has only around 48 people, a 0.2 gap is well inside the normal wobble. So my honest wording is "no detectable improvement in interpretation", not proof of zero effect. A larger sample or a repeat measure would be needed to say more.

Verdict I would give: "The programme got people into the tools but barely improved how they read results. Keep the hands-on sessions, replace the lecture blocks with worked interpretation cases, and re-measure in one quarter. We do not have evidence yet that decisions improved."

Trade-offs and pitfalls

• Non-participants are never a perfect comparison, so state that plainly.
• A gain that fades by month three means the programme taught recall, not habit.
• Tool usage is easy to measure and easy to overvalue. Lead with interpretation and decisions.
• Say what you could not measure (decision quality) rather than implying it.

Direct answer

Onboard them in tiers (consume dashboards, then query, then build), give each tier a short guided path with safe defaults, and make self-serve (people answering their own data questions without filing a ticket) safe by design: curated and certified datasets (tables the data team has documented and tested and marked as trusted), least-privilege access (each person gets only the access their role needs), one canonical definition per metric, and cost guardrails (limits on how much a single query can spend). Watch whether people succeed without filing a ticket and keep coming back, not whether they attended a session.

What "safe" means

• Access and prerequisites: request through a self-service form, granted by role (RBAC, role-based access control). Sensitive columns (PII, personally identifiable information) are masked (replaced with placeholders, for example an email shown as j***@example.com) or excluded from the shared layer.
• Certified datasets: a small set of documented, tested tables marked "trusted" in the data catalog (the searchable inventory of datasets, owners and definitions). Raw tables stay hidden by default.
• Canonical metrics: one written definition and owner per metric (for example "active customer = at least one paid order in the last 30 days"), so two teams cannot get two answers.
• Guardrails: query cost limits, timeouts, and read-only access.

The onboarding path

1. Kickoff message and prerequisites (communication plan): who it is for, what they can do after, links to the catalog and support channel.
2. Self-study path: 60-90 minutes of short modules (catalog tour, how to read a dashboard, your first query).
3. Starter templates and sample queries: copy-and-edit queries on certified tables, so nobody starts from a blank page.
4. Skills checklist to sign off: finds a dataset in the catalog, states a metric's definition, runs a filtered query, and knows when to ask for help.
5. Office hours weekly, plus a data catalog onboarding checklist for each new dataset owner. Example lines:
   • Owner: a named person or team, for example the finance data team.
   • Description: one sentence on what a row means, for example "one row per paid order".
   • Freshness: how recently the data is updated, with a promise, for example "refreshed by 6 a.m. daily, alert if late".
   • Tests: automated checks, for example "order_id is never empty and never duplicated".

What tells you it is working (illustrative pilot)

Say 20 people are invited. Track:

• Activation: 12 of the 20 ran a successful query within 5 working days is 60 percent.
• Retention: how many of those 12 ran another query in the following 4 weeks. If 9 of the 12 did, retention is 9 / 12 = 75 percent; if only 3 did (25 percent), the training got people started but nothing brought them back.
• Ticket mix: count "how do I find X" questions falling, and "wrong number" complaints, which point to definition problems.
• Duplicate metrics created (should go down) and query cost per user.
• Time to first successful query and satisfaction from a two-question survey.

Gaming: logins and course completions are easy to inflate; count successful repeat queries instead. Also watch for people building unofficial copies of metrics, which means the certified layer is missing something they need.

Pitfalls

• Opening raw access "for speed" then spending months cleaning conflicting dashboards.
• Training without templates: attendance high, usage low.
• No owner for support, so answers come back late and people revert to asking the data team.

Direct answer

I would design the programme backwards from decisions: identify who makes which data-driven decisions, teach each group only what those decisions need, use short practical formats tied to their own dashboards, and measure whether decisions improved using leading indicators plus a comparison between groups trained earlier and later. I would be explicit that "decisions improved" cannot be proven by a survey alone.

1. Who learns what (segment by decision, not job title)

Common misinterpretations to cover: the denominator (per visit or per person), percent versus percentage points, averages hiding the spread, small samples, and confusing correlation with cause.

2. Formats

• Short live workshops (45 minutes) on the audience's own data.
• Self-serve tooling onboarding (tools people use without asking an analyst): a guided first dashboard task and a data catalogue (a searchable list of available datasets and metric definitions) tour.
• Champions and office hours, plus mentorship pairing between analysts and business staff.
• Metric templates so every new metric has a name, definition, numerator, denominator, window, owner and caveats. Filled in: Name: repeat purchase rate. Numerator: customers with 2 or more paid orders. Denominator: customers with 1 or more paid order. Window: trailing 90 days. Owner: growth analytics. Caveat: excludes gift cards.
• For product teams, an experiment template: hypothesis (a testable guess, for example "a shorter checkout form raises completed orders"), primary metric (the one number that decides success, for example completed orders per visitor), guardrail metric (a metric that must not get worse, for example refund rate), planned sample size (how many users you need to see a real difference, for example 10,000 per group) and duration (for example two full weeks), decided before starting.

3. Cadence

A quarterly cycle: assess, teach the top two gaps, practise on a real decision, review.

4. How you know decisions improved

• Leading indicators: fewer disputes about metric definitions, fewer repeat ad hoc data requests (one-off questions sent to the data team), more self-serve queries, more experiments with a stated hypothesis.
• Decision-quality check: sample recent decision memos (randomly pick 30 from each cohort's memos, without choosing favourites) and score each 0 to 2: 1 point if it cites a defined metric by name, 1 point if it states what result would change the decision. Two reviewers score independently and compare.
• Comparison: train one cohort first and use the not-yet-trained cohort as a comparison, then train the second.

Worked example (illustrative numbers set for the example)

Cohort A (60 people) is trained in quarter 1. Cohort B (60 people) is trained in quarter 2. After quarter 1, we randomly sample 30 decision memos from each cohort's memos (as in the check above): 15 of A's 30 cite a defined metric, and 9 of B's 30 do.

A: 15/30 = 50%     B: 9/30 = 30%
difference = 50% - 30% = 20 percentage points   (not "20 percent")

Here "cites a defined metric" means the memo names the metric as defined in the catalogue (for example "repeat purchase rate"), not just "sales are up". The gap suggests training helped, but it is not conclusive: with only 30 memos per group the 95% margin on a 20-point gap is roughly plus or minus 24 points (standard error of about 12 points), so I would sample more memos (all 60 per cohort narrows it to about plus or minus 17) before claiming an effect. The groups may also differ (confounding: for example, cohort A may be a more data-savvy team that would have written better memos anyway), so I would confirm when B is trained: if B rises toward A's level in quarter 2, that supports the effect.

Trade-offs and pitfalls

• Attendance and quiz scores prove nothing about decisions.
• Training without tooling fails: people need a trustworthy, well-defined dashboard to practise on.
• Do not try to teach everyone everything. Depth is for champions.

Direct answer

Interviewing an expert one analyst at a time does not scale because the knowledge stays in the expert's head and in each analyst's private notes. I would convert the expert's know-how into shared, maintained assets (a glossary of business terms, a log of the quirks that explain the numbers, and definitions embedded in the data layer, meaning the tables and views that queries are built from), using structured sessions where analysts capture the knowledge and the expert only reviews it. The test of success is a new analyst who can answer common domain questions without booking the expert.

Step 1: find what to capture first

Mine the last few months of analyst questions (Slack, tickets) to rank recurring domain questions. Capture those first; they cost the experts the most time.

Step 2: structured elicitation (tacit knowledge is what experts know but rarely say)

• Shadowing: an analyst sits in a real month-end close (the monthly process where finance finalises the books) and records the steps and judgment calls.
• "Walk me through this number" sessions: the expert explains one metric from source to report while the session is recorded and an analyst writes it up.
• Ask about exceptions: "What always surprises new people about this number?"

Step 3: turn it into durable assets

Step 4: keep it alive

The expert reviews for 15 minutes rather than authoring from scratch; entries carry an owner and a review date tied to the close cycle; domain office hours and a short embedded rotation (an analyst spends a week with finance) refresh the knowledge.

Worked example (a quirks-log entry)

Why Jan revenue on the dashboard differs from finance
- Definitions: bookings = total value of contracts signed; billings = amount invoiced to the customer; recognised revenue = the share of that money the company has earned so far under accounting rules.
- Annual contracts are invoiced at signing but recognised evenly over 12 months.
- Example: a 12,000 annual contract signed on Jan 1.
  Bookings in January: 12,000. Billings in January: 12,000. Revenue recognised in January: 12,000 / 12 = 1,000.
  Difference between billings and revenue in January: 11,000.
- Rule: dashboards labelled "revenue" must use recognised revenue. Billings and bookings are separate metrics with their own labels, never called "revenue".

An analyst who reads this no longer needs to ask why the two reports disagree, and the same rule can be encoded as a data test (an automated check, as defined above) that fails if a "revenue" metric is computed from invoice date, so the trap is caught without anyone remembering the note.

How to measure it

• Expert interruptions per week (fewer over time).
• A check: give a new analyst five common domain questions and see how many they answer from the assets alone.

Trade-offs and pitfalls

• Asking experts to write documentation themselves rarely happens; let analysts write and experts correct.
• Docs that are not tied to the data layer drift away from what the queries do.
• What would change my call: if one expert holds unique, high-risk knowledge and is leaving soon, do intensive recorded sessions first and clean them up afterward.

Direct answer

Repeated lessons mean learning stays local: nobody owns turning an experience into something reusable, and nothing pushes it to where the next team's decisions are made. I would build a lightweight loop: capture from real events, store in one searchable place with an owner, and inject the lessons into the places where work is already reviewed (onboarding, design review, templates), then measure whether the same failure stops recurring.

1. Capture (small and routine)

• Recurring failure modes: from incident reviews (postmortems, meaning structured write-ups of what went wrong and how to fix it) and project retrospectives, tag each with a failure-mode category (a short label for the type of failure, for example: retry-storm, config-drift, missing-monitoring, data-leakage). A retrospective is a team meeting at the end of a project to discuss what to keep and change.
• Good practices and solution patterns: after each customer engagement or project, a short reflective review (what worked, what surprised us, what I would template) that feeds changes to shared templates.
• Keep the entry format small so people actually write it: context, problem, what worked, when not to use it, owner, last reviewed date.

2. Own and store

A rotating curator (or a small guild, meaning a volunteer group of practitioners from different teams who look after one craft) keeps the catalog; each entry has a named owner and a review date (for example twice a year) so old advice is retired. One home, findable by search.

3. Get it used (the hard part)

• Onboarding: teach the top failure modes in the first week.
• Design and architecture review: add checklist items derived from the catalog ("does this design retry without limits?"), so lessons reach decisions.
• Templates: bake practices into project, PR, and service templates so the good way is the default.
• Incentives: credit for adopting or contributing a pattern; make citing the catalog a normal part of a design doc.

4. Example: a cross-team ML best-practice programme

Goals: cut repeated data-leakage and evaluation mistakes. Formats: a monthly show-and-tell (a short informal talk where one team demos what it learned), a shared model-review checklist, and shared evaluation templates. The three checklist items in plain words: leakage (the model accidentally saw information it would not have at prediction time, such as the answer, so test scores look better than reality); baseline (a simple comparison model, so you know whether the fancy one actually helps); eval set integrity (the test data was never used for training or tuning). Before/after case: a team reports 96% accuracy, the checklist asks 'was any feature created after the outcome?', they find one, remove it, and the honest figure is much lower but real. Ownership sits with a rotating ML guild lead.

Worked example (illustrative)

Three teams each shipped a bug where unlimited retries caused duplicate charges. Entry created:

Failure mode: unbounded retries cause duplicate side effects
Context: payment and email calls behind a queue
What worked: idempotency key per request, retry cap of 3 with backoff
When not to use: read-only calls
Owner: Platform guild, last reviewed 2026-09

In plain words: an idempotency key is a unique ID sent with each request so the receiving service recognises a repeat and does not charge twice. A retry cap of 3 means try at most 3 times, and backoff means waiting longer between tries (1s, 2s, 4s). Without a cap and a key, a slow payment service causes the caller to retry, and each retry that succeeds charges the customer again: one purchase, three charges.

Then: add a design-review checklist question, mention it in onboarding, and add the idempotency key to the service template. Measure: incidents tagged with this failure mode per quarter, and how many design docs cite the entry. If the tag count falls, the loop works; if the entry is cited but the tag count stays flat, the guidance is not effective and needs rewriting.

Trade-offs and pitfalls

• A catalog nobody is pushed to use is a graveyard. Reach into existing review steps rather than asking people to visit a new site.
• Too much process makes people stop writing entries; keep it a 10-minute task.
• What would change my call: if the problem is really that teams do not trust central advice, seed the catalog with lessons written by practising teams, not by a central group.