Data-Driven Business Decision-Making Questions

Direct answer

I turn a vague "should we?" into a decision with a clear yes/no test: what would have to be true for this to be worth doing? I break that into a handful of questions, answer each with the cheapest evidence available, and stop researching when the next piece of information could not change my recommendation. With almost no prior analysis, two weeks of focused work is usually enough to give a conditional recommendation.

Structured elaboration

1. Write the decision down. "Should we spend $500k a year to sell to a new customer segment (mid-size healthcare companies), yes or no, by the end of next month?" Include the size of the commitment and the deadline.

2. Build an issue tree. An issue tree breaks one big question into parts that do not overlap. Four branches usually cover it:

• Is the segment big enough?
• Can we win there (does our product fit, who are the competitors)?
• Can we serve it (support, compliance, integrations)?
• Do the economics work?

3. Ask "what would have to be true?" For each branch, state the minimum belief needed to say yes, for instance "we can close at least 17 deals a year". Then pick the cheapest evidence for each: existing data pulls, 8 to 10 customer calls, a competitor scan, a talk with sales.

4. Decide when you know enough. Stop when (a) each branch is rated green, yellow or red with a one-line reason, (b) no red remains without a plan, and (c) the next study would not change the yes/no. Say how confident you are.

Worked example

Illustrative economics: average contract value $40,000 a year, 75% gross margin, so each deal contributes $30,000. The initiative costs $500,000 a year, so break-even is 500,000 / 30,000 = 16.7, rounded up to 17 deals a year.

At 10% win rate, 200 prospects gives 20 deals, above 17 but tight. That is yellow, and I would say so: recommend a limited 6-month trial, with a check at month 3 that at least 6 deals are in late stage.

Trade-offs and pitfalls

• Starting with the answer you want and collecting only supporting facts. Include one test that could kill the idea.
• Researching forever. Set the date first; research expands to fill the time.
• If calls reveal a compliance cost much larger than assumed, the recommendation flips to no or to a different entry (partner first).

Direct answer

With little data, build the simplest funnel that connects the partnership to money, use ranges instead of single guesses, find the break-even point, and recommend a short, capped pilot rather than a full commitment. Communicate the result as "most likely, plausible low, plausible high, and what we would need to see to proceed".

Step 1: Simple model (all numbers illustrative)

Customers = partner audience x click-through x conversion
Net = customers x revenue per customer x (gross margin - partner revenue share) - fixed setup cost

Base case: audience 400,000; click-through 2% gives 8,000 visitors; conversion 5% gives 400 customers. Revenue per customer $300 a year, gross margin 60%, partner revenue share 20%, so contribution per customer is 300 x (0.60 - 0.20) = $120. Fixed integration cost $30,000.

Net = 400 x 120 - 30,000 = $18,000.

Step 2: Scenarios

Break-even is 30,000 / 120 = 250 customers, which is 62.5% of the base case.

What assumptions matter most

The two funnel rates, click-through and conversion, because the partner's audience is not ours and we have no history with it. Contribution per customer and fixed cost are things we can largely look up or negotiate. With limited data, borrow proxies: our own past campaigns to similar cold audiences, and any numbers the partner shares from other collaborations, treated cautiously.

Communicating the uncertainty

"Our best estimate is about +$18k in year one, with a plausible range from -$16k to +$71k. The most we lose is about the setup cost of $30k, since a bad outcome is mostly capped at that, while the upside in the high case (+$71k) is about 2.4 times that cap. The payoff is asymmetric, but it is not lopsided enough to skip the pilot. The case works if we win at least 250 customers."

Recommendation

Sign a 60-day pilot with minimal integration. Proceed to the full deal if the pilot brings in at least 42 customers (250 x 60 / 365 is about 41, rounded up), assuming a steady rate. If conversion is below that, renegotiate terms or stop.

Note what the 42-customer bar means: it is only the pro-rata break-even, not a margin of safety, and it gives no cushion against the slower first-year ramp named in the pitfalls. At a true pace of 41 customers per 60 days, the chance of seeing 42 or more is close to a coin flip (Poisson, roughly 46%), so a result near the bar settles nothing. Treat roughly 42 to 60 customers as inconclusive and extend the pilot; proceed on a clear pass at or above the base-case pace of 400 x 60 / 365, about 66 customers, or when the shortfall is explained and the terms are renegotiated.

Pitfalls: presenting only the base case; counting the partner's total audience as reach; ignoring that a first-year ramp is slower than the steady state; treating the partnership's strategic value as proof without numbers.

Direct answer

Treat the instrumentation change (a change in how events are recorded) as a break in the time series (the metric tracked week by week). First measure how much of the movement in the metric is the measuring instrument rather than customers. Then restate the numbers on a like-for-like basis, attach plain disclaimers where you cannot fully correct, and add controls so the next change does not surprise anyone.

1. Quantify the effect

• Find the exact change date from release notes and from the data itself (a step in event volume on one day).
• Best method: an overlap period. If the old and new tracking both ran for some days, compare them on the same traffic. The ratio is your correction factor.
• If there was no overlap: compare against an independent source that did not change (server logs, payments, orders in the transaction database) for the same days, before and after.
• Last resort: model the trend with a step at the change date (a one-time jump in the level of the metric on that day), using unaffected segments or metrics as a control (a comparison group whose tracking did not change), and report the step as an estimate with a range. Illustration: web checkout went from 3.0% to 3.6% at the change, while the mobile-app checkout, whose tracking did not change, went from 3.0% to 3.1% over the same weeks. The 0.1 point is ordinary drift, so the estimated tracking step on web is about 0.6 - 0.1 = 0.5 points. I would report "0.5 points, plausibly 0.3 to 0.7" rather than a single figure.

Worked example (illustrative numbers)

Week 7 of 13 switched checkout tracking from a browser pixel (a small script in the shopper's browser that reports a purchase) to server-side events (purchase reports sent from our own servers). Browser pixels miss some purchases because ad blockers and privacy settings stop the script from running, while server-side reports are not blocked, so switching alone makes the same real sales look like more conversions. Reported conversion was 3.0% in weeks 1 to 6 and 3.6% afterwards, an apparent jump of 0.6 points. In weeks 5 and 6 the orders database (unchanged) implies a true rate of 3.5%, while the pixel showed 3.0%. So the old tracking captured 3.0 / 3.5 = 86% of conversions. Correcting the early weeks: 3.0 / 0.857 = 3.5%. The like-for-like change is 3.6 - 3.5 = 0.1 points, not 0.6. Most of the "improvement" was the instrument.

2. Adjustments and disclaimers

• Restate pre-change weeks on the new basis, or analyse only the post-change period.
• Show the raw and adjusted series together, with a vertical line at the change date.
• State the disclaimer in one sentence: "Weeks 1 to 6 are adjusted using a correction factor of about 0.86, estimated by comparing the old tracking with the orders database in weeks 5 and 6 (if the old and new tracking had run side by side, the factor would come from that overlap); the true figure could differ by a few tenths of a point."
• Say which conclusions survive. If the recommendation holds even at the pessimistic end of the correction, say so. If it flips, say the evidence is not yet strong enough.

3. Prevent it going forward

• A tracking change log that analysts must be told about before release.
• Run old and new tracking in parallel for a defined overlap before cutover.
• Automated checks that alert on sudden shifts in event volume or in the ratio to a stable source.
• Ownership: one named person approves instrumentation changes that touch reported metrics.

Pitfall: do not defend the original recommendation by waving at the VP's concern. The skeptic is right that the data changed; your job is to size it.

Direct answer

For a skeptical, time-poor, risk-focused executive team I structure the narrative as: the decision I need, the evidence in one picture, the risks and how I contain them, and what happens if they say no or wait. I treat their skepticism as a requirement, so I answer the likely objections before they ask, and I choose the one form of evidence that this audience trusts. Credible means they can check it; actionable means the last slide is a decision with an owner and a date.

Structured elaboration

Narrative order (answer first):

1. The ask and the recommendation (the first 2 minutes): "Approve X by Friday."
2. Why now: what it costs to wait.
3. Evidence: the strongest one or two facts, not everything I found.
4. Options considered: including doing nothing, so they see this is not the first idea I had.
5. Risks and containment: the downside in numbers, and a limit or stop condition.
6. What I am unsure about, and what would change my recommendation.

Anticipate objections with a pre-mortem. A pre-mortem means imagining it is a year later and the recommendation failed, then listing why. Each reason is an objection to answer in advance. I also talk to the likely skeptic privately beforehand (a pre-wire) and fold in their concerns.

Choose the evidence form that this audience trusts. For a launch-timing decision like this one that is rarely a dashboard. It is one comparison: for example revenue or retention of customers who use a capability versus those who do not, tied to dollars at stake, with the method stated in a line. Rankings and trade-offs beat raw numbers.

Worked example

Illustrative: recommend delaying a flagship (most important, most visible) feature launch by 6 weeks to fix onboarding drop-off (new customers abandoning the first-time setup steps). A 10-minute slot:

Four objections I would prepare for, with the words I would say:

1. "Is the data reliable?" Say: "It comes from the product event logs for the last 90 days, counted by account. I checked 20 accounts by hand and they matched the dashboard."
2. "Why not launch and fix in parallel?" Say: "The same setup step blocks the new feature, so we would launch into the problem. We only have two engineers for the fix, and a first impression with new customers is hard to repeat."
3. "What if the fix does not work?" Say: "We check at week 3. If setup completion has not improved, we stop the delay there, launch the feature without waiting out the remaining three weeks, and keep fixing onboarding alongside it."
4. "What are we giving up?" Say: "Six weeks of revenue from the launch. Here is that number next to the revenue we lose from accounts that never finish setup."

Trade-offs and pitfalls

• Too much evidence reads as insecurity; too little reads as opinion. One strong chart plus a one-line method is the balance.
• If a risk-focused leader sees the downside only in an appendix, trust drops. Show it in the main flow.
• The recommendation changes if the week-3 checkpoint is missed, and I say so before they ask.

Direct answer

Lead with the decision you want and the range of outcomes, not with the method. Executives need three things: what you recommend, how sure you are in money terms, and what protects them if you are wrong. Uncertainty is not an apology; it is information that sets how much to commit now.

Structured elaboration

1. Frame the hypothesis in one sentence. A hypothesis is the claim you tested and could be wrong about: "Offering a 10% loyalty discount to at-risk customers will keep enough of them to pay for itself."

2. Quantify uncertainty in business terms. Replace "p-value" and "confidence interval" with outcomes: a low, base and high case in dollars, plus the cost of being wrong. Say what drives the spread, so executives can see which belief matters most.

3. Recommend actions with guardrails. A guardrail is a limit set in advance: a spend cap, a stop condition and a check date. It turns "uncertain" into "bounded".

4. Plan the slide. One idea, a headline that states the conclusion, one visual showing the range, and a small box for assumptions and next step.

Worked example

Illustrative numbers. At-risk customers are those the team flags as likely to leave, for example because their usage has fallen for two months. A loyalty discount for them would cost about $600k a year at full rollout. Net annual impact is retained revenue (revenue from customers who stay because of the discount) minus the discount cost:

The three cases come from retained revenue of $0.3M, $1.5M and $2.2M, each less the $0.6M cost (0.3 - 0.6 = -0.3; 1.5 - 0.6 = +0.9; 2.2 - 0.6 = +1.6). The spread is driven by one belief: how many discounted customers would have stayed anyway.

Slide wording:

Headline: Pilot the loyalty discount on 10% of at-risk customers before rolling out

Likely outcome: +$0.9M a year (range: -$0.3M to +$1.6M)
Cost of being wrong: about $9k over the 8 weeks, never more than $60k
Why uncertain: we cannot yet tell who would have stayed without the discount
Guardrails: cap pilot spend at $60k; stop if retention lift is under 2 points at week 8
Decision today: approve the 8-week pilot

Visual: a horizontal range bar from -$0.3M to +$1.6M with a marker at +$0.9M and a vertical line at $0. The picture shows the whole range and the break-even line at once. Skip the dense table of model coefficients.

The pilot figures reconcile: 10% of the $600k full-rollout cost is $60k, the most the pilot population could cost in a full year, so $60k is the hard spend cap. The pilot itself runs only 8 weeks, which is 8/52 of a year, about $9k. The slide shows the expected loss ($9k) and the ceiling ($60k).

Trade-offs and pitfalls

• Do not hide a wide range inside a single "expected" number; executives feel misled when the low case arrives.
• Do not bury the recommendation behind caveats. Say what you would do under this uncertainty.
• Avoid statistical jargon on the slide. Keep the technical detail in an appendix for the one person who asks.