Business Acumen and Commercial Context Questions

Direct answer

Expected profit per 100,000 predictions is $175: the extra conversions are worth $375 and serving the model costs $200. Compare two model sizes by running the same calculation for each (value of the extra conversions, minus serving cost, minus any conversions lost to slower responses) and picking the higher expected profit per prediction, after checking that the difference is bigger than your measurement error.

The calculation, step by step

Definitions: lift is the increase in conversion probability the model causes (here from 0.5% to 0.75%, a lift of 0.25 percentage points). The conversion is worth $1.50 (treated here as profit per conversion; if $1.50 is revenue, multiply by the margin first). Every prediction costs $0.002 whether or not it converts.

1. Extra conversions: 100,000 x (0.0075 - 0.005) = 250.
2. Value of extra conversions: 250 x $1.50 = $375.
3. Serving cost: 100,000 x $0.002 = $200.
4. Expected profit: $375 - $200 = $175.

The 500 conversions that would have happened anyway (100,000 x 0.5%) are not credited to the model: only the lift is. Crediting all 750 conversions would give $1,125 - $200 = $925 and overstate the value fivefold.

Break-even: a prediction pays for itself if lift x $1.50 >= $0.002, so lift >= 0.00133, or 0.133 percentage points. This model clears that by a factor of about 1.9 (0.25 / 0.133).

Comparing model sizes

Per 100,000 predictions, profit = (lift x $1.50 x 100,000) - (cost per prediction x 100,000) - latency penalty. The latency penalty is the conversion lost because a slower model makes the page slower. It must come from an experiment, not intuition. In this illustration it is assumed to be 0.02 percentage points of conversion for each extra 50 ms, measured from the small model's 40 ms.

(The medium model's penalty is 0.02 x (90 - 40) / 50 = 0.02 pp, and the large model's is 0.02 x (200 - 40) / 50 = 0.064 pp.)

Reading the table:

• The large model loses money: its extra $0.008 per prediction over the small model buys at most 0.2 percentage points more lift, worth $0.003 per prediction against $0.008 more cost.
• Medium versus small depends on the latency assumption. Ignoring latency, medium earns $200 against $175, a clear win. With the penalty it earns $170 against $175, a tie. The right move is to measure the latency effect in an A/B test before paying for the bigger model.
• Choose the small model, because the gain from medium is inside the uncertainty of the inputs and small is cheaper and faster. Revisit if the value per conversion rises, since a higher value shifts every row toward bigger models.

Trade-offs and pitfalls

• Cost is incurred on every prediction, but the benefit only appears on those that change an outcome. A model is a good business when the benefit per prediction exceeds the cost per prediction, not merely when accuracy rises.
• Lift must come from a randomised comparison against no model (or the previous model). Offline accuracy gains do not translate one-for-one into conversion lift.
• Include fixed costs (training, monitoring, on-call) in a full comparison. Per-prediction cost alone ignores them.
• Compare on the same basis: per 100,000 predictions on both sides, with the value per conversion held constant.

Direct answer

A strong story has this shape: a specific project and a specific sceptical audience, evidence the audience could check for itself (the money at stake, a baseline, a small pilot), uncertainty handled by giving a range and a staged commitment with a stop rule, and an outcome that includes what did not go to plan. The composite example that follows is written the way I would tell it in the room (the figures are derived from the stated inputs, not measured results from a real company).

Situation

I was the data scientist on a business-to-business software product. Customer success leaders (non-technical) were losing accounts without warning, and I wanted funding to build a churn-prediction model (a model that scores accounts by how likely they are to cancel). Finance and the customer success vice-president were wary: past "AI projects" had run long and never reached daily use.

Evidence I used

1. The size of the problem, from their own data. 2,000 accounts at about 10% annual churn is 200 cancellations a year. At an average of $6,000 revenue per account per year, that is $1.2M of annual revenue lost.
2. A baseline they could trust. I back-tested a simple model (ran it on last year's data as if it had been live, then checked against what really happened) and showed it would have flagged 300 accounts of which 120 actually churned: precision (share of flagged accounts that churned) 40%, recall (share of all churners caught) 60% of 200.
3. A conservative value model. If the team saves 25% of the flagged accounts that would have churned (the save rate: the share of at-risk accounts it actually keeps), that is 30 accounts, or $180,000 of revenue, which at an 80% gross margin (the share of revenue left after the direct cost of serving it) is $144,000 of gross profit. Outreach to 300 accounts at 1.5 hours and $80 per hour costs $36,000, so net benefit is $108,000 a year.
4. A range, not a point. Save rate 20% to 30% gives net benefit of $79,200 to $136,800 a year. The save rate was the assumption I flagged as the one I could not know yet.

How I handled the uncertainty

I did not ask for the full build. I asked for an 8-week pilot costing about $40,000 with two stop rules (results agreed in advance that end the project) measured over the eight weeks: if the model flagged fewer than half of the accounts that later churned, or the customer success team could not save at least 20% of flagged accounts in a live trial, we would stop. A pilot at the base case pays back in about 4.4 months of benefit ($40,000 / $108,000 x 12), and at the 20% save rate in about 6 months. I also told them which risks I could not remove: the model might flag accounts the team could already tell were at risk, in which case the value is lower.

The words I used in the room: "Here is the problem in your own numbers: about 200 cancellations a year, $1.2 million of revenue. I am not asking you to trust a model. I am asking for $40,000 and eight weeks to test it on last year's accounts and then on live ones. If it misses half the accounts that go on to cancel, or your team cannot keep at least one in five of the accounts it flags, we stop, and you have spent $40,000 instead of funding a full build. At a 25% save rate it is worth about $108,000 a year after costs; at 20%, about $79,000. Which of those do you think is wrong?" Inviting the challenge to the save rate is what moved them: the customer success lead's own experience with similar outreach became the middle of the range.

Result and what I learned (the shape to aim for)

State plainly what happened, including the shortfall. In this composite the first month of the live trial saved 15% of the flagged accounts that would have churned, below the 25% base case, because outreach reached them late. We moved the scoring job to Monday morning and re-measured: weeks five to eight saved 27%. Over the full eight weeks the save rate was 21%, just above the 20% stop rule. At 21% the net benefit is 120 x 0.21 = 25.2 accounts x $6,000 x 0.8 - $36,000 = about $85,000 a year, a payback on the $40,000 pilot of about 5.7 months, so the sponsors approved the next stage using 21% as the planning figure, not 25%. The honest close names the lesson: the stakeholders agreed because the ask was small, reversible and measured on their numbers, and because I had told them in advance what result would make me stop.

Pitfalls

• Opening with model architecture: non-technical sponsors fund outcomes and risk limits, not methods.
• Giving a single confident ROI (return on investment, the net benefit relative to the cost) figure. It collapses the first time one input moves.
• Telling a story with no cost to you or no thing that went wrong. Interviewers read that as polish over substance.
• Real numbers matter: use your own project's figures, and only claims you can source.

Direct answer

Precision is the share of the cases the model flags that are truly positive: true positives (real cases the model correctly flagged) divided by (true positives plus false positives, which are wrong flags). Recall is the share of all real positives the model catches: true positives divided by (true positives plus false negatives, which are real cases the model missed). I favour precision when a wrong flag is expensive, for example automatically suspending a seller's account; I favour recall when a miss is expensive and a wrong flag is cheap, for example sending suspected payment fraud to a human reviewer. Which way to lean follows from the ratio of the two error costs, and the model's score threshold is how you move along that trade.

Two production scenarios

Favour precision: automatic suspension of seller accounts for suspected policy violations. A wrongly suspended seller loses sales for days, files appeals, burdens support and may leave the platform; call that $400 per false positive. A violating listing that stays up one more day costs about $60. A false positive is roughly seven times the cost of a miss, so the model should flag only when sure.

Favour recall: flagging payment fraud for human review. A missed fraud costs $135 (the lost goods or money plus a chargeback fee, which is the charge the bank levies on the merchant when a cardholder disputes a payment). A false alarm costs a reviewer a couple of minutes plus a little customer friction, about $20. A miss is roughly seven times a false alarm, so the model should flag generously.

Worked comparison (illustrative)

Assume 100,000 items a month with 1,000 true positives, and three operating points. An operating point is one setting of the score threshold (flag everything scoring above it); sliding the threshold traces a curve of precision and recall pairs for the same model, and each row below is one setting on that curve:

(Each row follows from the two ratios and the 1,000 real positives. For High precision: caught = recall x 1,000 = 0.50 x 1,000 = 500; missed = 1,000 - 500 = 500; alerts raised = caught / precision = 500 / 0.80 = 625; false alarms = 625 - 500 = 125.)

The best point flips: high precision is cheapest for suspensions ($80,000 against $546,000 for high recall), and high recall is cheapest for fraud review ($40,500 against $70,000 for high precision).

Operational and financial consequences

• Precision-first (the suspension scenario): at the high-precision point only 125 sellers are wrongly suspended, but 500 violating listings stay up. Those misses cost 500 x $60 = $30,000 a month, spread over many small $60 misses, while the 125 wrongful suspensions cost 125 x $400 = $50,000. Nothing needs a review queue here because suspension is automatic.
• Recall-first (the fraud-review scenario): only 100 frauds are missed, but 2,250 alerts need reviewing (against 625 at the high-precision point), with cost and delay in the queue. In the fraud case, going from the middle point to high recall saves only $3,250 a month ($43,750 against $40,500) while adding 1,000 alerts, so if reviewers cannot clear 2,250 alerts I would choose the middle point.

Pitfalls

• Choosing by F1 or accuracy without pricing the errors.
• Ignoring review capacity, which sets a ceiling on how many alerts you can raise.
• Using one threshold for all customers when error costs differ across segments.
• Forgetting that the best point shifts when costs or fraud patterns change.

Direct answer

A strong answer is a short story with five parts: the business problem and the baseline (the number before the change), the metric I chose and why, what I did and what it cost, how I measured the effect, and the limits on what I could claim. Below is a model story whose numbers are all derived from assumptions shown in it; a real answer replaces them with your own.

Model story

Situation and baseline. A subscription product charges $30 a month. About 20,000 renewals a month; 4% fail on the first attempt (800 failed payments). The existing retry logic recovered 50% (400), so 400 customers a month were lost to payment failure, worth $12,000 a month in revenue.

Metric choice. I chose recovery rate of failed renewals as the main metric because the change could move it within weeks and finance understands it. Overall churn was too slow and noisy to attribute. As a guardrail (a number that must not get worse), I tracked support contacts about duplicate charges.

Action. I changed when retries happen, spacing them toward typical paydays and sending a card-update email before the second retry.

Measurement. A four-week randomised test (customers assigned to a group by chance) with a control group (customers left on the old logic, for comparison) and a treatment group (customers given the new retry timing). Control: 200 of 400 recovered (50%). Treatment: 232 of 400 (58%). The difference is 8 points. The standard error (the typical size of the sampling noise in the difference) is sqrt(0.5 x 0.5 / 400 + 0.58 x 0.42 / 400) = sqrt(0.001234) = 3.5 points. An approximate 95% interval is the difference plus or minus 1.96 standard errors: 8 +/- 1.96 x 3.5 = 8 +/- 6.9, so 1.1 to 14.9 points. Read it as: the true gain is plausibly anywhere in that range.

Value and cost.

• 8 points x 800 failures = 64 more recovered renewals a month x $30 = $1,920 revenue, x 80% gross margin (the share of revenue left after direct delivery costs) = $1,536 contribution (profit after those costs) a month.
• Cost: three engineering weeks at $4,000 = $12,000 once, plus $200 a month for the email service.
• Net $1,336 a month, so payback is $12,000 / $1,336 = 9.0 months.
• Across the interval: using the rounded interval ends, at 1.1 points: 0.011 x 800 x $30 x 80% = $211, less the $200 service cost, is a net of about $11 a month (no practical payback); at 14.9 points: 0.149 x 800 x $30 x 80% = $2,861, less $200, is $2,661 a month (payback 12,000 / 2,661 = 4.5 months).

Limits on attribution. (1) 400 customers per group is small; the honest range is wide. (2) I counted only the recovered renewal payment; the extra months those customers stay were not measured, so I did not claim them. (3) Four weeks may not represent paydays across seasons. (4) A test shows the effect on this customer base now, not for a different market or price.

How I explained it to a non-technical stakeholder

"We changed when we retry failed cards. In a four-week test, 58 of every 100 failed renewals were recovered instead of 50. That is worth about $1,500 a month of contribution, $1,300 after running costs, and pays back in about nine months. The true gain is somewhere between roughly nothing and $2,700 a month, so we are extending the test." Visual: one chart with the two recovery rates as dots with their intervals (so the overlap is visible), and a small payback line showing the three cases, rather than a table of rates.

What separates strong answers

• A baseline stated before the change, not reconstructed afterwards.
• A comparison group, or a stated reason why none was possible.
• Cost included, not only benefit.
• An explicit statement of what you did not claim.
• A decision that followed (extend the test, ship, stop).

Direct answer

I would not decide from either metric alone. I would translate both into one comparable number (revenue per user over a horizon, adjusted for retention), test with a controlled experiment, and only then ship. A feature that raises engagement and lowers revenue per session can still be good if sessions rise by more than revenue per session falls, and bad if the extra engagement cannibalises revenue (replaces sessions that would have earned more) or shortens users' lifetime with the product.

The identity that settles most of the argument

Revenue per user = sessions per user x revenue per session.

Illustrative result from an A/B test (users randomly assigned to old or new experience), per week:

Break-even: revenue per session can fall by up to 1 - 4.0/4.5 = 11.1% before revenue per user drops. Check this is not only first-week novelty (people trying a feature because it is new): engagement bumps often fade.

Time horizon

• Short term (1 to 4 weeks): revenue per user and guardrails (metrics that must not get worse, such as errors, support contacts and refunds).
• Medium term (8 to 12 weeks): retention of the treated cohort, since engagement without retention is not a gain. Keep a small long-term holdout (for example 5% of users deliberately stay on the old experience for a quarter, as a comparison) to measure the delayed effect.
• Long term: lifetime value (LTV, total margin one user is expected to bring). A simple steady-state estimate is monthly ARPU (average revenue per user) x gross margin (the share of revenue left after the direct cost of serving it) / monthly churn (share of users lost each month). Dividing by churn works because 1 / 5% = 20: with 5% leaving each month, the average user stays about 20 months, so lifetime margin is 20 months x ($20 x 70% = $14 a month) = $280.

Choosing between a short-term revenue lift and a retention gain

Illustrative base: ARPU $20 a month, 70% gross margin, 5% monthly churn, so LTV = 20 x 0.7 / 0.05 = $280.

A loses despite the revenue gain: with +3% ARPU, churn can rise only to 5% x 1.03 = 5.15% before LTV falls. Segment the result (new vs established users, plan, region); a feature that helps new users and hurts loyal ones needs a different decision from one that moves everyone slightly.

Reconciling teams that optimise different goals

Suppose one team owns weekly active users and another revenue per user. Do not split the difference by negotiation. Agree on a shared decision rule before the test:

1. A shared top-line, such as revenue per user and retained users together (revenue = users x revenue per user).
2. Each team's metric becomes a guardrail on the other: the engagement team may not ship if revenue per user falls by more than an agreed amount; the revenue team may not ship if retention falls.
3. When the metrics conflict, the decision is made by whoever owns the shared top-line, using the LTV estimate and its stated assumptions.
4. Record the assumptions (horizon, margin, churn) so the decision can be revisited when results arrive.

Decision and pitfalls

Ship if treatment revenue per user is up by more than the noise (the confidence interval, the plausible range for the true effect, sits entirely above zero, so the gain is unlikely to be chance), retention is flat or better at the holdout checkpoint, and no key segment is harmed. Otherwise iterate on the design. Pitfalls: judging on weeks of novelty; mixing gross and net revenue (if a discount pushes gross revenue per session to $1.00 but refunds and discounts hand back $0.08, net is $0.92, so compare like with like); adding the two teams' percentage gains instead of multiplying (+12.5% sessions and -8% revenue per session do not make +4.5%; 1.125 x 0.92 = 1.035, which is +3.5%); ignoring cost to serve the extra sessions (support, servers and payment fees that each session adds).