Data-Driven Business Decision-Making Questions

Short answer

I would not roll it out to everyone yet. A 3% lift in signups is a measure of how many people start; support tickets and angry comments are early signs that the ones who start may not stay. I would check whether the extra signups are worth having, fix the friction, and widen the rollout in steps, moving on only when agreed results (gates) are met.

Evidence to look at

• Quality of the extra signups: activation and early retention of new-flow users versus old-flow users, in the same period.
• Ticket detail: tickets per 100 signups, what they are about, which step they come from.
• Comments: themes, and whether they come from converted users or abandoners.
• Segments: are some groups hit harder?
• Cost: support cost, and any effect on refunds or cancellations.

Weighing the lift against the hidden cost (illustrative)

Activation means a new user completes the first meaningful action that shows they got value (for example, sending a first message or finishing setup). Relative lift means the percentage change from the old rate: 10.0% x 1.03 = 10.3%.

Per 1,000 visitors: old flow converts 10.0% (100 signups); new flow 10.3% (the 3% relative lift: 103). Suppose 40% of old-flow signups are still active at 90 days, and each retained customer is worth $240 (an assumed lifetime revenue per retained customer).

• Old: 100 x 0.40 x $240 = $9,600.
• New, if retention falls to 36%: 103 x 0.36 x $240 = $8,899.

Break-even retention for the new flow is 40 / 103 = 38.8%. A fall of just 1.2 points cancels the lift. Tickets cost far less: the old flow generates 6 tickets per 100 signups and the new flow 9 per 100, at $10 each. Old: 100 signups x 6% = 6 tickets. New: 103 x 9% = 9.27 tickets. The extra 3.27 tickets cost about $33 per 1,000 visitors. The real risk is that the lift is converting people who leave.

When behavior and attitudes disagree

What people do (conversion, retention, task success) shows outcomes; what they say (comments, ratings) shows why and what may follow. They can diverge both ways. Here behavior is up while comments are negative. In the reverse case, task performance drops while stated satisfaction rises, from novelty or a nicer look. I weight behavior for outcomes, and use attitudes to diagnose causes and predict the longer term; comments can overrepresent the vocal minority (the small share of people who bother to write in), so I size them against tickets per signup.

Short-term versus long-term

A lift that shows up in one week and retention that shows up in three months are on different clocks (one is known within days, the other only after months). I give more weight to the long-term effect when the decision is hard to undo or the customer lifetime is long.

What I tell leadership

"The lift is measured (I am assuming the test shows it is beyond chance), but we have not shown it is worth having. Keep the new flow where it is, fix the top friction step, and expand when early activation is at least as good and 90-day retention is at or above 39%, just over the 38.8% break-even."

Use early activation as a leading indicator (an early signal that predicts the later outcome), since 90-day retention takes three months.

A caution on the 39% gate: 90-day retention is a noisy measure at this scale. With 1,000 signups per arm its standard error is about 1.5 points, and telling 40% from 38.8% (a 1.2-point drop) with 95% confidence and 80% power needs roughly 26,000 signups per arm. So 39% is a planning line, not something a small rollout stage can resolve. I ask for the interval on the retention difference rather than a single number, require its low end to stay above the break-even gap (a drop of 1.2 points), and keep a holdout on the old flow during expansion. The margin is also thin: at exactly 39% the new flow earns 103 x 0.39 x $240 = $9,641 per 1,000 visitors against $9,600 for the old flow, almost no gain, so the case for shipping really rests on retention staying near 40%.

Direct answer

I do not average the three sources. I ask what each one actually measures, how it could mislead, and how close it sits to the decision (will customers use and pay for this?). Behavior beats opinion, and our own customers beat the market in general. Then I recommend a small, time-boxed bet that is designed to settle the disagreement, not a full build and not a "no".

Structured elaboration

Step 1: classify each source by what it can and cannot tell you.

The weights are a judgment call, so I write them down and show them to stakeholders. That is what makes the call open to challenge instead of hidden.

Step 2: check sample bias and effect size. Sample bias means the people measured are not the people you care about. Effect size means how big the difference is, not just whether one exists. A small effect in a biased sample should barely move the decision.

Step 3: look for a story that explains all three. For example, interviews love the idea but usage is flat, because the people who talk to us are power users.

Step 4: recommend a hybrid validation plan with a pre-agreed pass/fail bar, so the result decides the argument.

Worked example

Illustrative feature: scheduled report exports in a B2B analytics product.

• Internal: 1,200 of 8,000 weekly active accounts (15%) export data by hand every week, a workaround that signals need. Their 90-day retention is 82% against 74% for others, an 8-point gap. That gap is partly selection (engaged users do everything more), so I treat it as a hint, not proof.
• Analyst report: says automation is a top buying criterion in the category. Useful context, but it says nothing about our accounts.
• Interviews: 3 of 12 said they would pay. 3 of 12 is 25%, but with so few people a 95% confidence interval (the range the true share plausibly falls in) runs from roughly 9% to 53%. The interviews cannot tell us how many customers want it. Recruited from our largest accounts, they also over-represent one segment.

Reading: behavior supports real need (15% already work around the gap), the analyst report is consistent but weak, and interviews explain why but cannot size it.

Recommendation: do not fund a full build. Fund a 4-week prototype for 10 design-partner accounts (customers who agree to try early versions), spanning small and large accounts to fix the sample bias. Pass bar set in advance: at least 6 of 10 use it in 3 of the 4 weeks and at least 4 say they would pay at the planned price. Pass leads to a full build. Fail leads to dropping the idea and writing down why.

How much that bar proves: with only 10 accounts, the bar is a screen, not proof. Chance of at least 6 of 10 using it, computed with the binomial formula: if the true usage rate were 30%, about 5%; at 40%, about 17%; at 50%, about 38%; at 80%, about 97%. So a pass is unlikely if real usage is low, but a pass cannot size the demand, and a miss at a true 50% happens about 62% of the time, so a narrow miss (4 or 5 of 10) means extend the test rather than drop the idea. Design partners also volunteer, so they lean friendly; that is why the pass leads to a staged build with a further gate, and not to an unconditional full commitment.

Trade-offs and pitfalls

• The call flips toward a full build if a usage cohort comparison controlling for account size still shows the retention gap, or if a lost-deal review shows this feature costing us deals. It flips toward no if the export activity behind the 15% turns out to come from a handful of heavy accounts (for example, if the top 10 exporters produce most of the export volume), because then the need is concentrated in a few customers, not spread across the 1,200.
• How to run the cohort check: split accounts into size bands (say 1-20 seats, 21-100, 100+), and within each band compare 90-day retention of exporters with non-exporters. Controlling for account size means comparing like with like. Large accounts both export more (they have more data and more reporting duties) and renew more, so a raw gap can come from size alone. If the gap shrinks to near zero inside each band, exporting was a marker of big accounts, not a cause of retention: correlation, not cause. If it stays near 8 points in every band, the case for the feature is stronger.
• A lost-deal review means reading the recorded reasons on deals we lost in the last two quarters and counting how many name missing scheduled exports. Selection, as above, means the people in a group were not picked at random, so a difference between groups may come from who they are rather than from what they did.
• Pitfall: treating the analyst report as the tiebreaker because it looks authoritative. Pitfall: reading 25% of 12 interviews as a market share.
• Tell stakeholders the weights and the bar before the prototype runs, so nobody can move the goalposts afterwards.

Direct answer

Lead with the decision you want and the range of outcomes, not with the method. Executives need three things: what you recommend, how sure you are in money terms, and what protects them if you are wrong. Uncertainty is not an apology; it is information that sets how much to commit now.

Structured elaboration

1. Frame the hypothesis in one sentence. A hypothesis is the claim you tested and could be wrong about: "Offering a 10% loyalty discount to at-risk customers will keep enough of them to pay for itself."

2. Quantify uncertainty in business terms. Replace "p-value" and "confidence interval" with outcomes: a low, base and high case in dollars, plus the cost of being wrong. Say what drives the spread, so executives can see which belief matters most.

3. Recommend actions with guardrails. A guardrail is a limit set in advance: a spend cap, a stop condition and a check date. It turns "uncertain" into "bounded".

4. Plan the slide. One idea, a headline that states the conclusion, one visual showing the range, and a small box for assumptions and next step.

Worked example

Illustrative numbers. At-risk customers are those the team flags as likely to leave, for example because their usage has fallen for two months. A loyalty discount for them would cost about $600k a year at full rollout. Net annual impact is retained revenue (revenue from customers who stay because of the discount) minus the discount cost:

The three cases come from retained revenue of $0.3M, $1.5M and $2.2M, each less the $0.6M cost (0.3 - 0.6 = -0.3; 1.5 - 0.6 = +0.9; 2.2 - 0.6 = +1.6). The spread is driven by one belief: how many discounted customers would have stayed anyway.

Slide wording:

Headline: Pilot the loyalty discount on 10% of at-risk customers before rolling out

Likely outcome: +$0.9M a year (range: -$0.3M to +$1.6M)
Cost of being wrong: about $9k over the 8 weeks, never more than $60k
Why uncertain: we cannot yet tell who would have stayed without the discount
Guardrails: cap pilot spend at $60k; stop if retention lift is under 2 points at week 8
Decision today: approve the 8-week pilot

Visual: a horizontal range bar from -$0.3M to +$1.6M with a marker at +$0.9M and a vertical line at $0. The picture shows the whole range and the break-even line at once. Skip the dense table of model coefficients.

The pilot figures reconcile: 10% of the $600k full-rollout cost is $60k, the most the pilot population could cost in a full year, so $60k is the hard spend cap. The pilot itself runs only 8 weeks, which is 8/52 of a year, about $9k. The slide shows the expected loss ($9k) and the ceiling ($60k).

Trade-offs and pitfalls

• Do not hide a wide range inside a single "expected" number; executives feel misled when the low case arrives.
• Do not bury the recommendation behind caveats. Say what you would do under this uncertainty.
• Avoid statistical jargon on the slide. Keep the technical detail in an appendix for the one person who asks.

Direct answer

Do not pick a number by gut or average the two. Build a bridge (a line-by-line reconciliation from one figure to the other) until the gap is explained. Then stand behind finance's number as reported revenue (it comes from the closed, reviewed ledger), call the warehouse figure a different measure, and tell leadership how much of the gap you have explained. The exception: if the bridge shows finance missed something legitimate, say so and change your position.

Step 1: Pin the definitions

Two correct numbers can differ because they measure different things. Check: revenue recognized (earned under accounting rules) vs billed or booked; gross vs net of refunds, credits, tax and discounts; currency conversion date; time-zone cutoff for "the month"; which legal entities are in scope; and whether finance's figure includes month-end adjustments made after the warehouse loaded.

Step 2: Localize the gap

Compare the two by day, product and region. A gap spread evenly smells like a definition difference. A gap concentrated on one day or one product smells like a pipeline defect or a late adjustment.

Step 3: Test the usual suspects

• Refunds and credit notes in finance but not netted in the warehouse.
• Duplicate rows: an order joined to several payment rows is counted once per payment (a join fan-out).
• Timing: invoices dated in the month but recognized in the next one.
• Test or internal accounts left in the warehouse.

Worked example (illustrative numbers)

Finance reports $10.0M and the warehouse $11.0M, so the warehouse is 10% higher and the gap is $1.0M.

Removing the four explained items takes the warehouse to 11.0 - 0.95 = $10.05M, within $0.05M (0.5%) of finance.

What I tell leadership

"Revenue for the month is $10.0M, the finance close figure. The warehouse showed $11.0M. We have explained $0.95M of the $1.0M gap (95%); $0.05M is still unexplained and I will report back on it. I am highly confident in $10.0M as reported revenue. I am less confident in warehouse-based breakdowns by segment until the double-counting is fixed."

Confidence is stated as how much of the gap is explained plus the size of what remains, not as a vague "fairly sure".

Pitfalls and what would flip the call

• Never hide the unexplained piece inside an "other" bucket.
• Do not blame finance without evidence. If the bridge showed finance had excluded a legitimate entity, I would say finance's figure was incomplete and quote the corrected one.
• Afterwards, fix the warehouse model, add a monthly automated reconciliation check against the ledger, and publish one agreed definition of revenue.

Short answer

First separate the quality of the decision from the quality of the outcome. A good decision can have a bad result and a bad one a good result, so I audit what we knew and did at launch, then check whether the result is distinguishable from noise. In this case the biggest finding is likely a process flaw: without a comparison group, the program could not tell success from failure.

Step 1: reconstruct the decision

• What evidence supported launching: was the model validated, how precise was it?
• What outcome was expected, and was a success threshold written down?
• What options were considered: no program, a different intervention, a trial first?

Step 2: check the comparison

Flagged accounts are the ones the model thinks are at risk, so comparing them with unflagged accounts mixes selection (who the model picked) with effect (what the outreach did). The right comparison is flagged and contacted versus flagged and not contacted. A holdout is a group deliberately left alone so it can serve as that comparison; a randomized holdout picks that group by chance, so the two groups are alike apart from the outreach. If no such holdout exists, say so plainly.

Decision quality versus outcome quality, in a small example

Decision A: launch with a written success bar and a randomized holdout, and the result is flat. The decision was good (it can tell us the truth); the outcome was disappointing. Decision B: launch with no holdout and churn happens to fall because the market improved. The outcome was good, but the decision was poor because we would wrongly credit the program. Judge the process, not the result.

Step 3: noise or real? (illustrative)

500 flagged accounts, 85 churned in three months: 17%. Suppose past data (before the program) says accounts scored this high churned about 18% of the time without outreach. That is where 18% comes from: the expected churn with no outreach.

The standard error (how much a percentage measured on 500 accounts typically wobbles by chance) is:

SE = sqrt(0.17 x 0.83 / 500) = 0.0168, about 1.7 points
95% interval = 17% +/- 1.96 x 1.7 points = about 13.7% to 20.3%

Read it as the range of true churn rates consistent with what we saw. The 18% baseline sits inside it, and so does 14%, a true 4-point saving. With 500 accounts we cannot tell "outreach did nothing" from "outreach helped a little".

How many accounts would we need? To reliably see a drop from 18% to 14% (95% confidence, meaning a 5% false-alarm rate, and 80% power, meaning an 80% chance of catching a real 4-point effect), the standard two-group sample-size formula gives:

n per group = (1.96 x sqrt(2 x 0.16 x 0.84) + 0.84 x sqrt(0.18 x 0.82 + 0.14 x 0.86))^2 / 0.04^2
            = about 1,300 accounts per group

So about 2,600 accounts in total, far more than we had.

Step 4: check each link in the process

What I change

• Keep a randomized holdout among flagged accounts, sized for the effect we care about.
• Write the success threshold and review date before launching.
• Measure outreach reach and timing as well as churn.
• Consider targeting accounts most likely to respond to outreach, not just most likely to churn.

Conclusion: the program may not work, or may work modestly; the process did not allow us to know. That is the decision flaw, not bad luck.