Prioritization and Trade-Off Decisions Questions

Why test. With low-confidence inputs, a score is a guess. Two items 4% apart are effectively tied, so I would test whether the order survives realistic error.

Setup (illustrative). The score is RICE: Reach x Impact x Confidence / Effort. A: Reach 3,000, Impact 1, Confidence 80%, Effort 3 person-months = 800. B: Reach 2,400, Impact 1, Confidence 80%, Effort 2.5 = 768. Gap: (800 - 768) / 768 = 4.2%.

Test 1: one-at-a-time sensitivity test. Change each input by a plausible amount and see if the order flips.

• B's reach 2,400 to 2,000: B drops to 640, A stays ahead.
• A's effort 3 to 4: A drops to 600, so B wins.
• A's reach 3,000 to 2,400: A drops to 640, so B wins.
  Break-even: A's effort only has to rise to about 3.1 person-months (800 x 3 / 768 = 3.125) for B to take the lead. That is tiny, so the ranking is not stable.

Test 2: ranges and simulation. Give each input a range instead of a single guess, then let the computer draw random values from those ranges 10,000 times and count how often A beats B. This is a Monte Carlo simulation (repeating a calculation with random inputs to see how often each outcome occurs). The ranges are wider than the setup point values to represent low-confidence estimates: A's reach 2,000 to 3,500 (point value 3,000), A's effort 3 to 4 (point value 3), B's reach 1,800 to 2,800, B's effort 2.5 to 3.5, and confidence 50% to 80% for both. Each random.uniform(low, high) call picks a random number between the two values.

import random
random.seed(7)
def rice(r, i, c, e):
    return r * i * c / e
wins = 0
N = 10000
for _ in range(N):
    a = rice(random.uniform(2000, 3500), 1, random.uniform(0.5, 0.8), random.uniform(3, 4))
    b = rice(random.uniform(1800, 2800), 1, random.uniform(0.5, 0.8), random.uniform(2.5, 3.5))
    wins += a > b
print(f"A beats B in {wins / N:.0%} of {N} draws")

Output: A beats B in 53% of 10000 draws. Read 53% as 'under these ranges the order is close to a coin flip', not as proof of a tie: the draws express our uncertainty, so even an A that is truly 30% better would win only about 81% of draws, and A would need to be about twice as good to win nearly all of them. Two caveats on the setup: the ranges are not centred on the point values (A's effort range of 3 to 4 only moves up from 3), and confidence is drawn independently for each item, whereas if both share the same uncertainty about confidence a shared draw gives about 54%. Either way the conclusion holds: the data cannot separate them.

What I do when it flips

• Do not decide on the score. Choose by a tiebreak (a rule used when scores cannot separate options): reversibility (how cheaply it can be undone), learning value (how much we learn by doing it), dependency on other work.
• Spend a day or two buying information on the input that matters most. Examples: a quick query counting how many users actually hit the affected screen (reach), a spike (a short time-boxed investigation) to firm up effort, or a fake-door test (show a button for a feature that does not exist yet and count clicks) to measure demand.
• If instrumentation (the tracking that records what users do) is incomplete, use a proxy, a measurable stand-in such as similar past launches, or a lightweight test.
• Ship the smaller or more reversible item first. Here B is 2.5 person-months against A's 3, so with no new data I would ship B first, and use the saved time to firm up A's reach.

What flips my call: new data narrowing a range.

Direct answer. With 50 uncertain options, do not run a heavy optimizer first. Narrow with simple rules, score the remaining options with ranges for the uncertainty, then pick the set with a ratio-based heuristic (a rule of thumb that is quick and usually good, not guaranteed best: here, rank by value per unit of cost), checking mutual exclusions and synergies by hand. Use formal optimization (integer programming, which finds the best combination under constraints) only when the set is large, constraints are many, and the input estimates are trustworthy enough to deserve that precision.

Steps

1. Cut first. Drop anything that fails a hard rule (no owner, no measurable outcome, or below an effort floor, a minimum size that is worth planning). In an illustrative case 50 might become 20 to 25; the real fraction depends on how many fail the rules.
2. Estimate with ranges. Give each initiative a low/likely/high value and cost, and a confidence. Uncertain value is shrunk (value times confidence) instead of trusted. For example, an initiative with a likely value of 8 points and 60% confidence is carried as 8 x 0.6 = 4.8.
3. Group mutually exclusive options (pick at most one per group) and mark synergies (A and B together are worth more than apart).
4. Treat dependencies as a graph: if X needs Y first, Y's value includes what it unlocks (for example, Y alone is worth 2 points for 2 person-months, but it unlocks X worth 7 for 4 person-months; judge Y+X together as 9 points for 6 person-months, a ratio of 1.5, instead of Y's standalone 1.0); sequence so value arrives early within the horizon, for example the quarter.
5. Select the set within monthly capacity, then test sensitivity: does the set survive if the top estimates are 30% worse?

Sensitivity means checking whether the choice survives if inputs are off. Steps 1 to 3 apply to every list. Steps 4 and 5 and formal optimization add precision when the stakes justify the extra effort.

Worked example (illustrative; capacity is 10 person-months in total, that is monthly capacity times the months in the planning horizon, for example 2.5 person-months a month over a four-month horizon; cost is effort in person-months, not calendar months; value in arbitrary points). A and B are mutually exclusive; C and D together earn a 3-point synergy bonus.

Greedy means take the best value-per-cost option first, then the next, until capacity runs out. Greedy by ratio picks C (3 person-months), then A (5, total 8), skips B because it is exclusive with A, then adds E (2, total 10): value 5 + 8 + 3 = 16 for cost 10. The best set (found by trying every allowed combination of the six items; no other allowed set within cost 10 beats 18, the next best being 17 for C, D, E, F) is B, C, D: 6 + 5 + 4 + 3 synergy = 18 for cost 10. The simple heuristic missed the synergy.

When a heuristic is better: when estimates are rough (the optimizer's precision is false), when the list changes monthly, or when stakeholders must understand the choice. When to optimize: many interacting constraints and a stable, well-measured value model. A good compromise is the heuristic plus a manual check of synergy pairs, which would have found B, C, D above.

Direct answer. Three "top priority" requests with no estimates cannot be ranked yet, so spend the first hours getting each to a comparable shape, write the gaps down as assumptions, then rank with a simple scoring pass and tell everyone it is a first cut. Example trio: a security fix, a search improvement and a marketing experiment.

What to ask each requester

• Value: who is affected, how many, and what number moves? What happens if it waits a month (cost of delay: the value lost for each unit of time the work is late)?
• Effort: has engineering seen it, and what is the rough size (days, weeks, a quarter)?
• Data: do we already have the data to measure it, or does measuring it need building first?
• Dependencies and deadlines: what else must happen first, and is there a real date (audit, contract, campaign slot)?

Assumptions to write down (examples): the security fix is sized at 2 weeks until the engineer confirms; the search gain is a guess from support tickets; the marketing test needs no engineering beyond tracking.

First-cut order in a day. ICE (Impact, Confidence, Ease, each scored 1 to 10; devised by Sean Ellis, whose published version averages the three scores, as in the table below) is quick enough for this. Some teams multiply the three instead; that gives the same order here: security fix 432, marketing experiment 270, search improvement 140. Illustrative scores:

Order: security fix, marketing experiment (cheap and quick to learn), then search. A hard-dated security issue would go first regardless of score. Send the order with the assumptions and say what would change it, then refine once estimates arrive.

Direct answer. Do not choose by instinct: spend the first day or two on cheap diagnostics that tell you which lever the model is starved of, then spend the rest of the two weeks on that lever. Hyperparameter tuning (adjusting model settings such as learning rate or tree depth) is usually last, because it rarely beats fixing data or features unless a quick check shows the model is sensitive to it.

Quick diagnostic (about one day)

1. Learning curve (a plot of score against how much training data the model saw): train on 25%, 50%, 75% and 100% of current labels and plot validation score. A curve still climbing steeply means more labels will help; a flat curve means they will not.
2. Train vs validation gap: a big gap suggests overfitting (the model memorized its training examples instead of learning the pattern), so more data or a simpler model; low scores on both suggests the model cannot express the pattern, so features.
3. Error analysis: read about 100 validation mistakes and tag each (ambiguous or wrong label, missing signal, rare class, noisy input). The biggest bucket names the lever.
4. Label-quality check: have a second person relabel about 50 examples and measure disagreement. A label is the answer attached to a training example.
5. Tuning sensitivity: run about 10 random configurations. If the best beats the default by less than the run-to-run noise from re-seeding, tuning is not your lever.

What "noise" looks like (illustrative). Re-seeding means retraining with a different random starting point. Train the default settings with 5 seeds and the validation accuracy comes out 0.795, 0.800, 0.803, 0.798, 0.805: a spread of 0.010. Ten random configurations whose best result is 0.808 beat the default (0.800) by only 0.008, less than the 0.010 seed-to-seed spread, so the gain is indistinguishable from luck and tuning is not your lever. A best of 0.830 would be a real signal.

Worked example (illustrative numbers). Validation accuracy 0.74, 0.78, 0.80, 0.81 at 25/50/75/100% of data: gains of 0.04, 0.02, 0.01, so returns are shrinking. Train accuracy 0.97 vs validation 0.81 is a 0.16 gap. Relabelling 50 errors, 12 (24%) had a wrong or ambiguous label. Decision: spend week one cleaning labels and writing guidelines, then retrain; spend week two on the one feature that the tagged errors most often lacked. More raw labelling is deferred because the curve is flattening and 24% of the problem is label quality, not label quantity.

Related trade-offs

• Data quality vs model accuracy: fix quality first; a better model fit to wrong labels learns the wrong thing.
• Retraining vs label quality: retraining on unchanged noisy labels reproduces the ceiling; relabel then retrain.
• Accuracy vs cost vs latency: compare each option by how much accuracy it buys per unit of inference cost (the compute spent each time the model makes a prediction) and added latency (response delay). For example, +1 point of accuracy that doubles serving cost from $2,000 to $4,000 a month is worth it only if that point is worth more than $2,000 a month. Bigger models or heavy features can win accuracy and lose the latency budget.

Flip condition. If the curve were still steep and relabel disagreement under about 5%, collect more labels first.

Direct answer. Convert both options into the same unit (cumulative revenue and subscribers over a horizon longer than the quick win), with the churn effect treated as an assumption to validate. On the illustrative numbers below I would choose the retention feature, and I would only flip to the revenue feature if cash runway is short or a holdout test shows its churn cost is much smaller than assumed.

Model (illustrative inputs, you replace with your own). 10,000 subscribers, $30/month, 400 new subscribers per month. Monthly churn (share of subscribers who cancel each month) is 3.0% today.

• Option A (revenue now): raises revenue per subscriber to $32 but lifts churn to 3.5%.
• Option B (retention): keeps $30 and cuts churn to 2.7%.

Each month: subscribers = previous x (1 - churn) + 400; revenue = subscribers x price. Traced for the first two months of A: month 1 subscribers = 10,000 x 0.965 + 400 = 10,050, revenue = 10,050 x $32 = $321.6k; month 2 subscribers = 10,050 x 0.965 + 400 = 10,098, revenue = $323.1k, so cumulative is $644.7k. For B, month 1 is 10,000 x 0.973 + 400 = 10,130 subscribers and $303.9k. Repeating this for 36 months gives cumulative revenue ($ thousands):

A leads for the first 18 months, B overtakes it at month 19, and by month 36 A is barely above doing nothing while B has about 13,017 subscribers against 11,032. Revenue-now features borrow from the future. A crossover chart plots both cumulative lines against month; the point where they cross (month 19 here) is the break-even that you show stakeholders.

Making the call.

1. Check the horizon: A wins on a 12-month view, B on 24. Ask what the business needs (cash, or compounding base).
2. Check the churn assumption. Run A as a holdout experiment (a randomly chosen control group that does not get the feature) and read churn, not just revenue.
3. Set guardrails (limits that must not worsen): churn, support tickets, refund rate. If a guardrail breaks, roll back.

Related situations.

• Free-to-paid versus paid retention. Weigh each by lifetime value (the total revenue you expect from one customer over the time they stay; roughly price / monthly churn, so $30 / 0.03 = $1,000 here). If 4% of free users convert, a free user converting is worth 0.04 x $1,000 = $40 on average. A retained paying user is worth the months saved x price: a feature that keeps a user two extra months is worth 2 x $30 = $60. Compare per engineering week.
• Fixed budget, onboarding versus search re-index. If both cost the same, take the one whose metric is closer to churn evidence. Early-life churn means cancellations in the first weeks: if 400 new subscribers join a month and 20% cancel within 30 days, 80 leave, and an onboarding fix that halves that keeps 40 more a month (about 40 x $30 = $1,200 of monthly revenue per cohort). Defer the other with a written trigger.

Communicating. Show the crossover chart and name the assumption that flips the call.