Knowledge Sharing and Team Enablement Questions

Direct answer

I would assume the problem is missing context, not bad intent, and give new PMs the engineering picture early, in their own language: what is cheap, what is expensive and why, in units of time and risk. Then I would change the process so constraints surface before a commitment, and offer options instead of a flat "no". The goal is that the PM comes to us with the platform constraint already in mind.

What I would do

1. Constraints one-pager for PMs. Not architecture, but "what this means for you": which changes are days, which are weeks, which are quarters, and the reason (for example "every customer-facing field passes through three services"). Tiers come from the engineers' own past work: look at the last 20 or so similar changes and see how long each took from start to release, then group them. A sample excerpt:

Add a field to an existing export ........ days   (one service, no data change)
Change how a field is stored ............. weeks  (schema change touches 3 services, needs a migration)
Make an export real time ................. quarter (reworks the pipeline; blocks other roadmap items)
Why: every customer-facing field passes through three services.

2. Early exposure. In their first month: a platform walkthrough, shadowing an on-call shift (the engineer who is paged when production breaks) or an incident review (a blameless meeting after an outage asking what happened and how to prevent it), and sitting in on estimation. Seeing a real failure teaches faster than a slide.
3. Options, not refusals. For a request, present two or three options with cost tiers and what each gives up. The PM makes the product decision; we make the cost visible.
4. Involve an engineer at discovery (the early stage where a PM works out what to build), before the roadmap commitment. A short "feasibility check" (an engineer's quick read on whether it is easy, hard or risky) step in the intake (the form or queue where requests arrive), a few days, not a gate.
5. Shared vocabulary. Explain tech debt (shortcuts taken earlier that make later changes slower) and coupling (parts of the system that depend on each other, so a change in one forces changes in the others) as "the reason this takes longer than it looks", with one real example. Illustrative: "Two years ago we hard-coded the customer ID format into four services to ship fast. Changing the format now means editing and releasing all four together, which turned a two-day request into a three-week one."

Worked example (illustrative)

A new PM asks for real-time custom fields on customer exports. Instead of "not possible", I offer:

The PM sees the cost curve and often picks A or B, then plans C as a later bet if customers justify it.

Measures (rather than a satisfaction feeling)

• Share of requests arriving with a feasibility check already done.
• Late-stage rework or scope reversals per quarter.
• The PM's own read after 90 days (a short survey).

Pitfalls

• Lecturing, or making PMs feel policed. Keep it collaborative.
• Hiding cost in engineering jargon. Translate into schedule and risk.
• Always saying yes to avoid friction, which pushes the cost downstream.
• What would change my approach: a PM repeatedly ignoring the constraints after the context is clear becomes a manager conversation, not a teaching problem.

Direct answer

Data literacy is the ability to read, question and use data sensibly: knowing what a number actually measures, reading a chart honestly, and knowing what data can and cannot support. It matters for a company that lives on dashboards because dashboards make numbers look authoritative, so a misread metric becomes a wrong decision at scale. I would teach first: what exactly does this number count? (the definition, denominator and time window), because every other skill depends on it.

What I would teach, in order

1. Metric definitions: what is counted, out of what, over which period.
2. Reading charts honestly: axes that start at zero or not, sample size, a single period versus a trend.
3. Averages versus the spread: a mean can hide a skewed distribution.
4. Correlation is not causation: two things moving together does not show one caused the other.
5. When to ask an analyst instead of deciding from the dashboard.

Worked example 1 (same data, different denominator)

A dashboard shows a 4% conversion rate. The company has 200 orders and 5,000 sessions (a session is one visit).

200 orders / 5000 sessions        = 0.04  = 4%
200 orders / 2500 unique visitors = 0.08  = 8%

The same 200 orders are 4% or 8% depending on whether you divide by visits or by people. Two teams quoting "our conversion rate" can disagree while both being right. Teaching people to ask "divided by what?" prevents that argument.

Worked example 2 (average versus typical)

Five orders: 20, 20, 25, 30 and 400. The sum is 495, so the mean is 495/5 = 99, but the median (the middle value when sorted) is 25. Reporting "average order value is 99" misleads, because four of five customers spent 30 or less.

Why it matters and pitfalls

• Non-technical staff make many decisions from dashboards, so their reading skill limits the value of the data team's work.
• Teach with the company's own dashboards and real recent decisions, not abstract statistics.
• Pitfall: turning it into a statistics course. Start with definitions and the common traps, and add depth only when a decision needs it.

Direct answer

One shared idea, three different jobs. I would teach the same core concept for about 15 minutes so everyone speaks the same language, then split the rest by what each audience must be able to do afterwards: executives decide and ask better questions, product managers apply it in specs, analysts use it hands-on. I choose the goal for each group first and build the activity backwards from it.

Structure of the slot (60 minutes, illustrative)

What each group leaves able to do

Sample shared core (15 minutes, for the ROC example below): "A model gives every case a score. We pick a cutoff: above it we flag the case, below it we do not. A high cutoff flags few cases and misses more real problems. A low cutoff catches more but raises more false alarms. In our fraud data, 100 of 1000 cases are truly fraud. At one cutoff we flag 200 cases and 80 are real fraud. Everything today is about who decides where the cutoff goes and what that costs."

The two worked examples below show the pattern (worked example A is the shorter illustration; B carries the full numbers).

Worked example A: a new semantic layer (a shared place where each business metric is defined once, so every dashboard uses the same definition). Executives: "which numbers can we trust to match across teams, and what do we ask for when they do not?" Product managers: "name the metric and the definition in the spec instead of describing a chart." Analysts: "add a metric to the layer and reuse it in a report."

Worked example B: interpreting ROC and precision-recall curves for business analysts. A ROC curve (receiver operating characteristic) plots the true positive rate (the share of real positives caught, the same number as recall) against the false positive rate (the share of real negatives wrongly flagged, for example 120 of 900 = 13.3%) as the score cutoff (threshold) moves. A precision-recall curve plots precision (of the items flagged, how many are right) against recall (of the real positives, how many are caught). Use a fixed example: 1000 cases, 100 real positives, 900 negatives.

Exercise 1: compute precision and recall for a threshold from the counts. Exercise 2: choose a threshold when a false alarm costs 1 unit and a miss costs 10 units. Cost = false alarms + 10 x misses: A = 120 + 200 = 320, B = 20 + 400 = 420, C = 310 + 100 = 410. Threshold A wins. Lesson: the threshold is a business choice, and with few positives the precision-recall view shows the false-alarm burden that a low false positive rate can hide. Look at row A: the false positive rate is only 13.3% (120 of 900 negatives), which sounds small, yet precision is just 0.40, so 6 of every 10 flagged cases are false alarms. That happens because negatives outnumber positives 9 to 1, so a small share of a big group is still a lot of cases.

The same concept for the other two audiences (using the same table).

• Executives (decision, 5 minutes): "At cutoff A we catch 80 of 100 frauds and review 200 cases; at B we review only 80 cases but miss 40 frauds. A miss costs about ten times a false alarm, so we choose A. Ask your team: what does a miss cost us, and what does a review cost us?" They leave able to set the cost ratio and ask for that trade-off instead of a single accuracy number.
• Product managers (spec): rewrite "the model should be accurate" as "flag at a level that catches at least 80% of fraud (recall) while keeping at least 40% of flags correct (precision); revisit if the miss-to-false-alarm cost ratio changes."

Pitfalls

• Teaching the mechanics to executives; they need the decision.
• Same slides for everyone with one Q&A; each group must practise their own skill.
• Ending with "any questions?" instead of a check that each group can perform the target action.

Direct answer

A question that keeps coming back is a symptom, so I would not just keep answering it. I would first sort the repeated questions by root cause: people disagree about the definition, people cannot find the answer that already exists, or the tooling makes the answer hard to get. Then I would fix the cause once, in the place where people actually ask, and measure whether the question stops coming back.

Step 1: diagnose from the evidence (a few days of work)

• Export two to four weeks of the Slack threads that ask about the metric. Read at least 30 to 40 of them; do not guess from memory.
• Tag each thread with one root cause:
  • Definition confusion: the asker has a number, but it does not match another number they saw (for example, two dashboards both call something "active users" and compute it differently).
  • Discoverability: a correct answer exists (a doc, a dashboard note), but the asker did not know where to look.
  • Tooling or access: the asker cannot self-serve, for example they lack dashboard access, the filter is confusing, or the data is stale.
  • Real data issue: the number is actually wrong. This is a bug, not a knowledge problem, and goes to the data owner.
• Also ask three askers directly: "Where did you look first, and what did you expect to find?" That reveals the path people really take.

Step 2: pick the durable fix that matches the dominant cause

Worked example (illustrative numbers)

Suppose 40 threads over two weeks asked about "monthly active customers". Tagging gives 22 definition confusion, 9 discoverability, 6 tooling, 3 real data issues (22 + 9 + 6 + 3 = 40, so 55% definition confusion). The dominant cause is definitions, so the fix is a single catalog entry:

Metric: Monthly active customers
Definition: distinct customers with at least one paid order in the trailing 30 days,
            excluding orders fully refunded
Grain: one row per customer
Refresh: daily
Owner: Analytics team (name + channel)
Known caveat: excludes free-trial accounts; the Growth dashboard's "active users" counts logins and is a different metric

Then link this from the tile tooltip, retire the duplicate tile, and reply to every future Slack question with the link (and if the link did not fully answer it, fix the entry the same day).

How I would know it worked

• Count of repeat threads per week on that metric (tagged), before and after. The target is a clear downward trend within a month or two, not zero.
• Time to first useful answer for a new analyst who has never seen the metric.
• Catalog page views from the tooltip (a sign people find it).

Trade-offs and pitfalls

• Writing more docs when the real problem is two conflicting definitions: the confusion returns.
• A catalog with no owner rots; give each entry an owner and a "last reviewed" date.
• Do not shame askers. A repeated question means the system failed, not the person.
• What would change my call: if most threads turn out to be real data issues, this is a data-quality problem and the fix is pipeline tests, not documentation.

Direct answer

Teach judgement, not tools. Non-technical staff do not need to know how a model is trained; they need to know what it is good at, where it fails, what must never be pasted into it, and how to decide whether an AI use is safe for their job. I would build one short shared core (90 minutes), role-specific labs for product, legal and sales built from their real decisions, and a one-page decision checklist. I would pilot it as a staggered rollout (the groups start training at different dates) with a waiting group as comparison, and set the success bar before the pilot starts.

Content: what each audience needs

Shared core concepts: a language model produces likely text from patterns, not verified facts; the same prompt can give different answers; quality must be measured on examples, not by demo; bias and privacy are risks; and "when not to use AI" is as important as "how".

Programme shape

• 90-minute core workshop with live examples of failures.
• 60-minute role lab where each group works its own scenarios.
• A one-page checklist: what data goes in, what happens if the answer is wrong, who reviews it, is it approved, how would we notice a failure.
• A monthly 15-minute open office hour with the AI team.

Pilot

Pick 30 people who reflect each function, not just volunteers (volunteers are already enthusiastic and skew results). Assign them to two cohorts (groups that go through the programme together) of 15 at random within each function, for example by drawing names, so that neither managers nor volunteers choose who trains first (a hand-picked cohort A would make any gain reflect who was picked, not the programme): cohort A trains now, cohort B trains four weeks later and serves as the comparison meanwhile. Both take the same 10 scenario questions (rephrased between rounds so memorisation does not help) before and after. Also track behaviours: unrealistic requests sent to the AI team, and legal review flags on AI-related material.

Worked example (illustrative numbers)

Scenario item: "A model summarises a contract and cites clause 14. What do you do before relying on it?" Correct: verify clause 14 in the source, because the citation may be invented.

• Cohort A: 4.0 out of 10 before, 6.5 after, a change of 6.5 - 4.0 = 2.5.
• Cohort B (not yet trained): 4.1 before, 4.4 at the same time, a change of 0.3.
• Effect over background change: 2.5 - 0.3 = 2.2 points.

Reading 2.2 points. On a 10-question test it means the trained group got about two more questions right than background change would explain. With only 15 people per cohort, averages wobble: if individual scores typically vary by about 2 points, an average of 15 people wobbles by roughly 2 / sqrt(15) = 0.5, and the gap between two such averages by roughly 0.7 points. So 2.2 is about three times that noise, which is convincing; a gap of 0.5 would not be.

Decide up front (a convention, not a law), for example: "scale if the trained group gains at least 2 points more than the waiting group and the behaviour measures do not worsen; if the gap is between 1 and 2, extend the pilot to more people; below 1, redesign".

Trade-offs and pitfalls

• Leaders often ask for "prompt tricks". Resist; tricks age in months, judgement lasts.
• Fear-heavy training makes people avoid useful tools, hype-heavy training makes them careless. Show both wins and failures.
• Content goes stale as tools change: assign an owner and review it every quarter.
• Scores measure knowledge, not behaviour, so keep the behavioural signals in the pilot.