Data-Driven Business Decision-Making Questions

Direct answer

I do not pick a winner among the three sources. I first find out why they disagree, because the disagreement is usually information: each source measures a different population, time or kind of behavior. Once one explanation fits all three, I decide on that basis, state my confidence in plain words, name the evidence that would change my mind, and commit to a decision date. Being decisive about the next step is different from claiming certainty about the facts.

Structured elaboration

1. Make a disagreement table. For each source write down who was measured, what was measured (stated intent or actual behavior), when, and who benefits from the answer.

2. Calibrate against history. Calibrating means scaling stated intent by how past stated intent turned out. If a past launch exists, compare what the survey predicted with what happened. That ratio is better than a generic discount.

3. Reconcile by segment and time. Often the sources agree inside one segment and diverge across segments.

4. Decide and communicate with calibrated language: "We think X (about 70% confident). The main risk is Y. We will know more by date Z, and if A turns out, we will switch."

Worked example

Illustrative: should we add a premium tier? "Top-box" means the share of respondents who chose the strongest answer on the scale ("definitely would buy"). To calibrate means to scale stated intent by how past stated intent turned out.

• Survey of 1,000 people: 60% "interested", 14% top-box (140 people).
• Partners: "customers do not ask for it."
• Sales: a comparable add-on (a different product from last year's add-on used for calibration below) sold in 450 of 5,000 deals, 9%.

Calibration: for last year's add-on the survey top-box was 20% and actual sales were 11%, so reality was 11 / 20 = 55% of the stated figure. Apply that to the new survey: 14% x 0.55 = 7.7%. That is close to the 9% in the sales data, so the survey and sales agree once the survey is discounted. This is a cross-check against a proxy product, not an independent measurement of the premium tier, and it applies one ratio (0.55) from a single past launch to every segment, although overstatement may differ between large and small accounts.

Now reconcile by segment. Splitting the same data:

In large accounts the calibrated survey (13.75%) and sales (14%) agree closely, so both point to real demand there. In small accounts both are low. The partners we asked sell mainly to small accounts, so "customers do not ask for it" is consistent with the data: they are describing the segment where demand is weakest, not contradicting the large-account signal.

Decision: launch the premium tier to large accounts first, not to everyone. Planning figure for large accounts: 12 to 14% uptake (the calibrated survey and sales numbers). I would say: "I am fairly confident (about two in three) that uptake in large accounts lands between 10% and 15%. That range is wider than the 13.75% and 14% estimates because the calibration rests on a single past launch. If the first 90 days show under 10%, we pause. Review on a set date." For a launch to everyone the same logic gives 7.7% to 9%, which is why I would not plan around a blended figure.

Trade-offs and pitfalls

• Averaging the three numbers hides the cause of the disagreement; the table finds it.
• Do not dismiss the partners because they disagree; check whether they see a segment you do not.
• To avoid sounding indecisive, give the decision first, the confidence second and the trigger that would change it third.

One-page pilot plan: guided setup checklist for new accounts

(Illustrative example of a product change; every number is a stated assumption.)

The ask: a 6-week pilot, with the go / change course / stop rule below agreed before we start.

What we test: half of new accounts get the checklist (treatment), half do not (control), assigned at random. One change only.

Metric: 14-day activation (an account completes its first key action within 14 days of signup). Baseline: 40%.

Why I expect it to work: new accounts today stall at setup; the checklist removes the step where most of them drop off. This is a hypothesis, which is why it is a pilot.

Range of impact I expect: from 0 to +4 points, most likely about +2 (a judgment from comparable onboarding changes; I will say if the evidence is thin).

Size and duration: to detect a 2-point difference (40% vs 42%) with 95% confidence and 80% power, a standard two-proportion calculation needs about 9,500 accounts per group, roughly 19,000 in total. Confidence of 95% means that if the checklist did nothing, we would be fooled into seeing a difference (in either direction) only about 1 time in 20; a false lift specifically, which is what the Proceed row requires, about 1 time in 40. Power of 80% means that if the true lift is 2 points, we would catch it 8 times in 10. The calculation:

n per group = (1.96 x sqrt(2 x 0.41 x 0.59) + 0.84 x sqrt(0.40 x 0.60 + 0.42 x 0.58))^2 / 0.02^2
            = about 9,500

1.96 and 0.84 are the standard values for 95% confidence and 80% power; 0.41 is the average of the two rates. A smaller true effect would need far more accounts, and a baseline of 25% instead of 40% needs fewer (about 7,500 per group). At 5,000 new accounts a week that is under 4 weeks of enrolment; I run 4 full weeks (20,000 accounts, 10,000 per group) plus 2 weeks for the 14-day window to complete: 6 weeks.

Decision rule (set now, not after seeing results)

(The 95% interval is the range of true lifts consistent with the data; "entirely above zero" means even its low end is a real improvement. A guardrail is a metric that must not get worse, defined under Guardrails below.)

Why +1.0 as the floor: it is about 50 extra activated accounts a week at 5,000 signups, roughly the size below which maintenance cost outweighs the gain (assumption).

Guardrails (things that must not get worse): support tickets per 1,000 new accounts up by more than 10% relative; 30-day retention of treated accounts lower than control by more than 1 point, with the 95% interval for the difference entirely below zero. A bare "below control" is not a breach: with about 10,000 accounts per group, chance alone puts the treated group below control about half the time even if the checklist does nothing, so that rule would stop the pilot on a coin flip. "Any guardrail breached" in the table means these thresholds.

Timing of the retention guardrail: 30-day retention needs 30 days after signup, so it is not complete at week 6 for accounts enrolled late. I read it on the first two weeks of enrolment (about 10,000 accounts) at about week 7 and on everyone at about week 9, and I do not release the full rollout until the first read is in. The activation read stays at week 6.

What would change the plan: if weekly signups fall well below 5,000, the pilot runs longer rather than being declared early.

Addressing the skeptic directly: the plan costs 6 weeks of one team's time, can be switched off at any point, and ends in one of three named outcomes, none of them "more analysis".

Short answer

I would not roll it out to everyone yet. A 3% lift in signups is a measure of how many people start; support tickets and angry comments are early signs that the ones who start may not stay. I would check whether the extra signups are worth having, fix the friction, and widen the rollout in steps, moving on only when agreed results (gates) are met.

Evidence to look at

• Quality of the extra signups: activation and early retention of new-flow users versus old-flow users, in the same period.
• Ticket detail: tickets per 100 signups, what they are about, which step they come from.
• Comments: themes, and whether they come from converted users or abandoners.
• Segments: are some groups hit harder?
• Cost: support cost, and any effect on refunds or cancellations.

Weighing the lift against the hidden cost (illustrative)

Activation means a new user completes the first meaningful action that shows they got value (for example, sending a first message or finishing setup). Relative lift means the percentage change from the old rate: 10.0% x 1.03 = 10.3%.

Per 1,000 visitors: old flow converts 10.0% (100 signups); new flow 10.3% (the 3% relative lift: 103). Suppose 40% of old-flow signups are still active at 90 days, and each retained customer is worth $240 (an assumed lifetime revenue per retained customer).

• Old: 100 x 0.40 x $240 = $9,600.
• New, if retention falls to 36%: 103 x 0.36 x $240 = $8,899.

Break-even retention for the new flow is 40 / 103 = 38.8%. A fall of just 1.2 points cancels the lift. Tickets cost far less: the old flow generates 6 tickets per 100 signups and the new flow 9 per 100, at $10 each. Old: 100 signups x 6% = 6 tickets. New: 103 x 9% = 9.27 tickets. The extra 3.27 tickets cost about $33 per 1,000 visitors. The real risk is that the lift is converting people who leave.

When behavior and attitudes disagree

What people do (conversion, retention, task success) shows outcomes; what they say (comments, ratings) shows why and what may follow. They can diverge both ways. Here behavior is up while comments are negative. In the reverse case, task performance drops while stated satisfaction rises, from novelty or a nicer look. I weight behavior for outcomes, and use attitudes to diagnose causes and predict the longer term; comments can overrepresent the vocal minority (the small share of people who bother to write in), so I size them against tickets per signup.

Short-term versus long-term

A lift that shows up in one week and retention that shows up in three months are on different clocks (one is known within days, the other only after months). I give more weight to the long-term effect when the decision is hard to undo or the customer lifetime is long.

What I tell leadership

"The lift is measured (I am assuming the test shows it is beyond chance), but we have not shown it is worth having. Keep the new flow where it is, fix the top friction step, and expand when early activation is at least as good and 90-day retention is at or above 39%, just over the 38.8% break-even."

Use early activation as a leading indicator (an early signal that predicts the later outcome), since 90-day retention takes three months.

A caution on the 39% gate: 90-day retention is a noisy measure at this scale. With 1,000 signups per arm its standard error is about 1.5 points, and telling 40% from 38.8% (a 1.2-point drop) with 95% confidence and 80% power needs roughly 26,000 signups per arm. So 39% is a planning line, not something a small rollout stage can resolve. I ask for the interval on the retention difference rather than a single number, require its low end to stay above the break-even gap (a drop of 1.2 points), and keep a holdout on the old flow during expansion. The margin is also thin: at exactly 39% the new flow earns 103 x 0.39 x $240 = $9,641 per 1,000 visitors against $9,600 for the old flow, almost no gain, so the case for shipping really rests on retention staying near 40%.

Short answer

I would not tell executives "ship" or "kill". I would recommend a staged launch with a holdout (a group deliberately kept on the old experience so we can measure the difference), with the reversal numbers agreed before launch. The revenue loss is a forecast and the retention gain is a hope, so the plan should buy evidence on the hope while capping the cost.

What executives need to see

Illustrative numbers: quarterly revenue is $10M, so a 3% cut is $300k. My own forecast has a range (say -1% to -5%, or $100k to $500k), so I show the range, not one number.

Pricing the uncertain benefit in business terms

Assume annual revenue of $40M. Each 1 point of extra annual retention keeps about 1% of $40M = $400k of annual revenue. Retained customers keep paying after the year they were saved, so the benefit accumulates. Suppose it builds up evenly (linearly) over 12 months, reaching the full $400k a year by month 12:

Year 1: average benefit is half of 400k = about 200k
Year 2: full 400k
24 months: 200k + 400k = 600k

So one point is worth about $200k in the first year and $600k over 24 months. So to recoup $300k:

• within 12 months, retention must improve by about 1.5 points;
• within 24 months, by about 0.5 points.

The decision therefore depends on the horizon leadership cares about, which I ask for explicitly. This ignores margin and discounting (valuing later money at less than money today), which I would add before the meeting.

This also assumes the 3% is a one-time dip in next quarter only, so I confirm with the product manager whether it continues. If the forecast is a run-rate that repeats every quarter, the loss is $300k a quarter, or $1.2M a year, and the bars move: recouping $1.2M within 12 months needs about 6 points (1,200k / 200k per point), and recouping $2.4M within 24 months needs about 4 points (2,400k / 600k per point). A gain that large is unlikely for a feature expected to cost revenue persistently, so in that case I would lean toward a small test or a redesign over a staged launch, and I would replace the 0.5-point bar below with the persistent-case figures.

Handling the uncertainty

Rank the evidence: (1) leading indicators (early signals that predict retention before it can be measured, such as engagement and repeat usage) from a real test, (2) outcomes of comparable past changes, (3) the product manager's reasoning. I give my range for the retention gain, an interval of plausible values (for example 0 to 2 points, most likely about 1) and compare it to the break-even bars above. Reversible does not mean free: a rollback can restore the feature but not the revenue lost meanwhile or any trust lost.

Limiting the downside

Launch to 20% of users first: the revenue cost is about 0.2 x $300k = $60k for the quarter, and we learn the real retention signal.

What would make me reverse

Agreed in advance:

• revenue loss in the treated group (the users who received the change) worse than the forecast's bad end (5%), checked at the end of the first quarter;
• after a full retention cycle (long enough to see most customers' renew-or-leave decision, for example 12 months for annual contracts), the interval for the retention gain sitting entirely below 0.5 points, since 0.5 points is what is needed to recoup the loss within 24 months. Until then the leading indicators guide whether to continue;
• a guardrail (a metric that must not get worse) such as complaints or cancellations rising.

If the signs are good (gain near or above 1 point, loss near forecast), widen the rollout in steps and keep a small holdout. Loss inside the forecast with a flat retention signal is a reason to stop, not to wait.

Short answer

First separate the quality of the decision from the quality of the outcome. A good decision can have a bad result and a bad one a good result, so I audit what we knew and did at launch, then check whether the result is distinguishable from noise. In this case the biggest finding is likely a process flaw: without a comparison group, the program could not tell success from failure.

Step 1: reconstruct the decision

• What evidence supported launching: was the model validated, how precise was it?
• What outcome was expected, and was a success threshold written down?
• What options were considered: no program, a different intervention, a trial first?

Step 2: check the comparison

Flagged accounts are the ones the model thinks are at risk, so comparing them with unflagged accounts mixes selection (who the model picked) with effect (what the outreach did). The right comparison is flagged and contacted versus flagged and not contacted. A holdout is a group deliberately left alone so it can serve as that comparison; a randomized holdout picks that group by chance, so the two groups are alike apart from the outreach. If no such holdout exists, say so plainly.

Decision quality versus outcome quality, in a small example

Decision A: launch with a written success bar and a randomized holdout, and the result is flat. The decision was good (it can tell us the truth); the outcome was disappointing. Decision B: launch with no holdout and churn happens to fall because the market improved. The outcome was good, but the decision was poor because we would wrongly credit the program. Judge the process, not the result.

Step 3: noise or real? (illustrative)

500 flagged accounts, 85 churned in three months: 17%. Suppose past data (before the program) says accounts scored this high churned about 18% of the time without outreach. That is where 18% comes from: the expected churn with no outreach.

The standard error (how much a percentage measured on 500 accounts typically wobbles by chance) is:

SE = sqrt(0.17 x 0.83 / 500) = 0.0168, about 1.7 points
95% interval = 17% +/- 1.96 x 1.7 points = about 13.7% to 20.3%

Read it as the range of true churn rates consistent with what we saw. The 18% baseline sits inside it, and so does 14%, a true 4-point saving. With 500 accounts we cannot tell "outreach did nothing" from "outreach helped a little".

How many accounts would we need? To reliably see a drop from 18% to 14% (95% confidence, meaning a 5% false-alarm rate, and 80% power, meaning an 80% chance of catching a real 4-point effect), the standard two-group sample-size formula gives:

n per group = (1.96 x sqrt(2 x 0.16 x 0.84) + 0.84 x sqrt(0.18 x 0.82 + 0.14 x 0.86))^2 / 0.04^2
            = about 1,300 accounts per group

So about 2,600 accounts in total, far more than we had.

Step 4: check each link in the process

What I change

• Keep a randomized holdout among flagged accounts, sized for the effect we care about.
• Write the success threshold and review date before launching.
• Measure outreach reach and timing as well as churn.
• Consider targeting accounts most likely to respond to outreach, not just most likely to churn.

Conclusion: the program may not work, or may work modestly; the process did not allow us to know. That is the decision flaw, not bad luck.