Data-Driven Business Decision-Making Questions

Direct answer

Take the CEO's enthusiasm seriously but do not let a weak report make the decision. Treat the report as a hypothesis (a claim to be tested), discount it for its weaknesses, and recommend a small, capped pilot with decision criteria agreed in advance. That turns "invest or not" into "invest if the pilot shows X".

1. Evaluate the report's claims

• Who produced and paid for it? A vendor or sponsor has a reason to show large gains.
• Method: was it a controlled comparison or just before-and-after? Are the "gains" averaged over success stories only (selection bias, where only winners are counted)?
• Sample and comparability: how many companies, how like ours (size, market, customer type)?
• Effect size and range: effect size is how big the improvement is. A single big number with no interval (a confidence interval: the range of true effects consistent with the data) is a warning sign.
• Does the mechanism make sense for our funnel, and is our own data consistent with it?

2. Structure the recommendation

1. Claim and why it is attractive. 2. What we can and cannot trust in the report. 3. Our own estimate of the opportunity, from our data, with a range. 4. The pilot and what it costs. 5. The decision rule. 6. The risk of waiting.

Worked example (all numbers illustrative)

The initiative costs $800k in year one and affects $20M a year of gross profit (revenue minus the direct cost of delivering it). The report claims a 12% lift ($2.4M). Break-even lift is the smallest gain that covers the cost: 0.8 / 20 = 4% of annual gross profit (both in $M). So the initiative pays back if the lift exceeds 4%, a third of the claim. That is useful: the CEO's case does not need the claim to be fully true, but we need evidence it is at least 4%.

3. Minimal-risk experiments

• Run a holdout test (some customers get the change, a comparable group does not) on about 10% of relevant traffic for about 8 weeks.
• Cap the pilot at $100k, which is 12.5% of the full cost, and make it reversible.
• Measure our own lift directly rather than trusting the report's metric definition.

4. Decision criteria: invest, hold or stop

Point estimate means the single best-guess lift the pilot measured; the interval is the 95% confidence interval around it, and its low end is the pessimistic edge.

The 6% bar is deliberately half the report's claim (a haircut: a cautious discount on a claim we cannot verify) and sits 2 points above the 4% break-even, a margin for error. The 2% floor on the low end is a judgment: it says the pessimistic edge must still be real, not zero. These are judgment calls for this example, to be agreed with the CEO before the pilot. For instance, a pilot measuring 5% with an interval of 1% to 9% falls under Hold: 5% is below 6%, the low end is below 2%, but the high end (9%) is above 4%, so it is neither a clear win nor a clear loss.

Can the pilot actually deliver that interval? The invest rule needs the low end of the interval to clear 2% when the point estimate is 6%, so the interval half-width must be about 4 points or less. Illustration with assumed inputs: if the pilot metric is a conversion rate of 4%, then for a relative lift the half-width is roughly 1.96 x sqrt(2 x (1 - 0.04) / (n x 0.04)) with n users in each group. Solving for 0.04 gives n of about 115,000 users per group. With about 30,000 per group the half-width is about 8 points, so a true 6% lift would usually land in Hold rather than Invest. Check the expected traffic in the 10% holdout over 8 weeks against this before agreeing the rule, and if it falls short, lengthen the pilot, widen the holdout, or use a less noisy metric. Otherwise the pilot cannot return a clear Invest, and a pass rule that is almost impossible to meet is a hidden Stop.

What would change my mind: a cheap, trustworthy internal analysis already showing a lift above break-even would justify skipping straight to a larger test.

Pitfall: telling the CEO the report is "wrong". Say it is unproven for us, and show the cheapest path to proof.

Short answer

First separate the quality of the decision from the quality of the outcome. A good decision can have a bad result and a bad one a good result, so I audit what we knew and did at launch, then check whether the result is distinguishable from noise. In this case the biggest finding is likely a process flaw: without a comparison group, the program could not tell success from failure.

Step 1: reconstruct the decision

• What evidence supported launching: was the model validated, how precise was it?
• What outcome was expected, and was a success threshold written down?
• What options were considered: no program, a different intervention, a trial first?

Step 2: check the comparison

Flagged accounts are the ones the model thinks are at risk, so comparing them with unflagged accounts mixes selection (who the model picked) with effect (what the outreach did). The right comparison is flagged and contacted versus flagged and not contacted. A holdout is a group deliberately left alone so it can serve as that comparison; a randomized holdout picks that group by chance, so the two groups are alike apart from the outreach. If no such holdout exists, say so plainly.

Decision quality versus outcome quality, in a small example

Decision A: launch with a written success bar and a randomized holdout, and the result is flat. The decision was good (it can tell us the truth); the outcome was disappointing. Decision B: launch with no holdout and churn happens to fall because the market improved. The outcome was good, but the decision was poor because we would wrongly credit the program. Judge the process, not the result.

Step 3: noise or real? (illustrative)

500 flagged accounts, 85 churned in three months: 17%. Suppose past data (before the program) says accounts scored this high churned about 18% of the time without outreach. That is where 18% comes from: the expected churn with no outreach.

The standard error (how much a percentage measured on 500 accounts typically wobbles by chance) is:

SE = sqrt(0.17 x 0.83 / 500) = 0.0168, about 1.7 points
95% interval = 17% +/- 1.96 x 1.7 points = about 13.7% to 20.3%

Read it as the range of true churn rates consistent with what we saw. The 18% baseline sits inside it, and so does 14%, a true 4-point saving. With 500 accounts we cannot tell "outreach did nothing" from "outreach helped a little".

How many accounts would we need? To reliably see a drop from 18% to 14% (95% confidence, meaning a 5% false-alarm rate, and 80% power, meaning an 80% chance of catching a real 4-point effect), the standard two-group sample-size formula gives:

n per group = (1.96 x sqrt(2 x 0.16 x 0.84) + 0.84 x sqrt(0.18 x 0.82 + 0.14 x 0.86))^2 / 0.04^2
            = about 1,300 accounts per group

So about 2,600 accounts in total, far more than we had.

Step 4: check each link in the process

What I change

• Keep a randomized holdout among flagged accounts, sized for the effect we care about.
• Write the success threshold and review date before launching.
• Measure outreach reach and timing as well as churn.
• Consider targeting accounts most likely to respond to outreach, not just most likely to churn.

Conclusion: the program may not work, or may work modestly; the process did not allow us to know. That is the decision flaw, not bad luck.

Direct answer

With little data I still recommend, but I recommend a bounded, reversible step instead of a full commitment. I make the assumptions explicit, size the pilot so that its result can actually change my mind, and set the spending cap and the stop rule before it starts. Thin data changes the shape of the decision (small, staged, checkable), not whether I give a view.

Structured elaboration

Act now or validate first? Four tests.

1. Reversibility: can we undo it cheaply? A pilot yes, a multi-year contract no. Reversible decisions can be made on thin evidence.
2. Cost of waiting: is a window closing (competitor, season)? If delay is cheap, gather more data first.
3. Value of information: would more data plausibly flip the decision? If every outcome of a study leads to the same action, skip the study. Example: if the segment is worth entering whether the study shows 5% or 15% conversion, the study has no value.
4. Cost of the test versus the cost of being wrong: do not spend $200k to avoid a $50k mistake. Applied here: a capped $150k pilot is reversible and the full launch is not, so the pilot passes tests 1 and 4.

State the assumptions the decision rests on. An assumption is something we believe but have not shown. Typically: customers in the new segment have the problem, will pay roughly what we charge, and can be reached at an acceptable cost. I rank them by how much the decision depends on each, and test the riskiest first.

Communicate the limits of the sample. Inconsistent feedback from 9 customers, 6 positive, is not "two thirds are positive". The 95% interval (the range of true shares that fit the data) for 6 of 9 runs from roughly 35% to 88%, which includes "barely half" and "nearly everyone". I say that plainly, and also say who the 9 were (who picked them, who is missing).

Worked example

Illustrative: a new small-business segment for a product sold to enterprises. No history, no benchmark, 9 mixed interviews.

Pilot design:

• Question: can we win qualified small-business leads at a rate that makes the segment worthwhile?
• Scope: 60 qualified leads over 8 weeks, with a separate small team and a hard spend cap of $150k.
• Success bar (set now): at least 6 of 60 convert (10%) and the sales cycle (time from first contact to signed deal) under 45 days. A qualified lead is a prospect who fits the target customer profile and has a real need and budget.
• Stop rule: if fewer than 3 convert by week 6, stop and write up why.
• Owner and check date: the segment lead, with a review at week 4.

Is 60 leads enough? This is a power question: power is the chance the pilot reaches the bar when the segment really is good. The binomial calculation (the standard way to count how often a given number of successes appears in 60 independent tries at a fixed rate) gives the chance of seeing 6 or more conversions: if the true rate were 3%, under 1%; at 5%, about 8%; at 10%, about 56%; at 15%, about 90%. Read it this way: if the segment is really poor (3%), a pass by luck is very unlikely, so a pass is meaningful evidence the rate is not tiny. If the segment is really at 10%, we still fall short of 6 about 44% of the time (100% minus 56%), so a miss is not proof of failure. I therefore treat a result of 3 to 5 conversions as "extend the pilot" and not as a verdict.

Check the week-6 stop rule the same way, because by week 6 only about three quarters of the leads (roughly 45 of 60) have been worked. At a true 10% rate, the chance of fewer than 3 conversions among 45 leads is about 16%, so the stop rule would wrongly end a segment that is exactly on the bar about one time in six. At a true 5% it fires about 61% of the time and at 15% only about 3%. That is acceptable for a spending guard on a $150k cap, but I say so: week 6 is a cheap-stop check for clearly poor results, and the launch decision waits for the full 60 leads at week 8.

Recommendation to leadership: approve the capped pilot, not the segment launch. Decision on launch at week 8 using the bar above.

Trade-offs and pitfalls

• The recommendation changes if a cheap, fast study (for example 10 more interviews with the right buyers) is available and could flip the decision; then validate first.
• Pitfall: a pilot too small to detect the effect you care about. Compute its power (as above) before running it.
• Pitfall: moving the success bar after seeing results.

Direct answer

I turn a vague "should we?" into a decision with a clear yes/no test: what would have to be true for this to be worth doing? I break that into a handful of questions, answer each with the cheapest evidence available, and stop researching when the next piece of information could not change my recommendation. With almost no prior analysis, two weeks of focused work is usually enough to give a conditional recommendation.

Structured elaboration

1. Write the decision down. "Should we spend $500k a year to sell to a new customer segment (mid-size healthcare companies), yes or no, by the end of next month?" Include the size of the commitment and the deadline.

2. Build an issue tree. An issue tree breaks one big question into parts that do not overlap. Four branches usually cover it:

• Is the segment big enough?
• Can we win there (does our product fit, who are the competitors)?
• Can we serve it (support, compliance, integrations)?
• Do the economics work?

3. Ask "what would have to be true?" For each branch, state the minimum belief needed to say yes, for instance "we can close at least 17 deals a year". Then pick the cheapest evidence for each: existing data pulls, 8 to 10 customer calls, a competitor scan, a talk with sales.

4. Decide when you know enough. Stop when (a) each branch is rated green, yellow or red with a one-line reason, (b) no red remains without a plan, and (c) the next study would not change the yes/no. Say how confident you are.

Worked example

Illustrative economics: average contract value $40,000 a year, 75% gross margin, so each deal contributes $30,000. The initiative costs $500,000 a year, so break-even is 500,000 / 30,000 = 16.7, rounded up to 17 deals a year.

At 10% win rate, 200 prospects gives 20 deals, above 17 but tight. That is yellow, and I would say so: recommend a limited 6-month trial, with a check at month 3 that at least 6 deals are in late stage.

Trade-offs and pitfalls

• Starting with the answer you want and collecting only supporting facts. Include one test that could kill the idea.
• Researching forever. Set the date first; research expands to fill the time.
• If calls reveal a compliance cost much larger than assumed, the recommendation flips to no or to a different entry (partner first).

Short answer

Compare the options on probability-weighted value per unit of the scarce resource (engineering capacity), and also look at how bad each bad case is. I would do B and C now, because they are cheap and still positive in their weak scenarios, and treat A as a stage-gated bet (funded in stages, with the next stage released only if the first one passes): pay a small amount to learn whether its big upside is real before committing its full cost.

Representing the uncertainty

Engineering capacity here is measured in the same $ thousand units as cost (the cost of an initiative is the engineering effort it takes). Replace each single-point estimate with two or three scenarios, each with a probability and a value. Expected value (EV) is the sum of probability x value: the average outcome if you could repeat the decision many times. The numbers below are illustrative (first-year value and engineering cost, same basis, in $ thousands):

Worked row for A: 0.30 x 1,500 + 0.30 x 400 + 0.40 x 0 = 450 + 120 + 0 = 570 EV; net EV = 570 - 400 = 170; EV per $1 of cost = 570 / 400 = 1.4 (about 1.4 back for each 1 spent); worst-case net = 0 - 400 = -400. For B, EV is 0.6 x 350 + 0.4 x 200 = 290, so 290 / 80 = 3.6. For capacity-limited decisions the columns that drive the recommendation are EV per $1 and worst-case net.

The probabilities are judgments, so I record who supplied them and invite the sponsor of A (the person advocating for it) to challenge them; I do not present them as facts.

Limited capacity changes the ranking

Say the team has 300 of engineering capacity this half (a half-year planning period). All three in full cost 510, and A alone costs 400, which is already more than the 300 available, so A in full does not fit this half at all. Funding it would mean raising capacity or dropping other work: at 400 of capacity, A would use all of it and displace both B and C, whose combined net EV is 275 (210 + 65) for only 110 of cost. That displaced 275 is the opportunity cost of A: the best alternative use of the same capacity. By net EV the order is B (210), A (170), C (65). By EV per dollar it is B (3.6), C (3.2), A (1.4). B is first on both lenses, and A is the only option with a real chance of losing everything.

Recommendation

1. Fund B and C now (110 of capacity).
2. Fund only stage 1 of A: a 60 prototype or pilot designed to reveal which scenario we are in. The remaining 340 of A's cost (400 - 60) is a later build. Because 340 is more than one half's 300 of capacity, that build would be spread across two halves or need added capacity, which is another reason to buy evidence first.
3. Revisit the 340 next half, only if stage 1 passes.

Why gating helps, assuming (optimistically) that stage 1 reveals the scenario perfectly. Success: build and keep the 1,500 value, having paid 60 + 340, so 0.3 x (1,500 - 60 - 340) = 330. Partial: the build returns 400 against 400 total spend, a net of zero, so 0.3 x (400 - 60 - 340) = 0 (we are indifferent between stopping and building). Failure: we stop after stage 1 and lose only the 60, so 0.4 x (-60) = -24. EV = 306, versus 170 for building A up front. A real pilot is noisier, so the gain is smaller, but it stays positive whenever stage 1 is informative and cheap relative to the build. Capacity used this half: 80 + 30 + 60 = 170 of 300.

What would change my mind

• Success probability of A: with the partial scenario held at 30%, ungated A matches B's net EV (210) at a success probability of about 33% (1,500p + 0.3 x 400 - 400 = 210). My estimate is 30%, so evidence that pushes it above 33% moves A up the queue.
• B's low scenario or cost: if B's cost doubles or its low case falls well below 200, the gap to A narrows.
• Capacity: with 510 or more available, do all three.
• Risk tolerance: if leadership cannot absorb A's -400 worst case, only the gated version is acceptable.

Telling A's sponsors

"A is not rejected. It is staged: here is the 60 test, here is the result that releases the next 340." That keeps the idea alive and ties its funding to evidence.