Prioritization and Trade-Off Decisions Questions

Direct answer. Do not choose by instinct: spend the first day or two on cheap diagnostics that tell you which lever the model is starved of, then spend the rest of the two weeks on that lever. Hyperparameter tuning (adjusting model settings such as learning rate or tree depth) is usually last, because it rarely beats fixing data or features unless a quick check shows the model is sensitive to it.

Quick diagnostic (about one day)

1. Learning curve (a plot of score against how much training data the model saw): train on 25%, 50%, 75% and 100% of current labels and plot validation score. A curve still climbing steeply means more labels will help; a flat curve means they will not.
2. Train vs validation gap: a big gap suggests overfitting (the model memorized its training examples instead of learning the pattern), so more data or a simpler model; low scores on both suggests the model cannot express the pattern, so features.
3. Error analysis: read about 100 validation mistakes and tag each (ambiguous or wrong label, missing signal, rare class, noisy input). The biggest bucket names the lever.
4. Label-quality check: have a second person relabel about 50 examples and measure disagreement. A label is the answer attached to a training example.
5. Tuning sensitivity: run about 10 random configurations. If the best beats the default by less than the run-to-run noise from re-seeding, tuning is not your lever.

What "noise" looks like (illustrative). Re-seeding means retraining with a different random starting point. Train the default settings with 5 seeds and the validation accuracy comes out 0.795, 0.800, 0.803, 0.798, 0.805: a spread of 0.010. Ten random configurations whose best result is 0.808 beat the default (0.800) by only 0.008, less than the 0.010 seed-to-seed spread, so the gain is indistinguishable from luck and tuning is not your lever. A best of 0.830 would be a real signal.

Worked example (illustrative numbers). Validation accuracy 0.74, 0.78, 0.80, 0.81 at 25/50/75/100% of data: gains of 0.04, 0.02, 0.01, so returns are shrinking. Train accuracy 0.97 vs validation 0.81 is a 0.16 gap. Relabelling 50 errors, 12 (24%) had a wrong or ambiguous label. Decision: spend week one cleaning labels and writing guidelines, then retrain; spend week two on the one feature that the tagged errors most often lacked. More raw labelling is deferred because the curve is flattening and 24% of the problem is label quality, not label quantity.

Related trade-offs

• Data quality vs model accuracy: fix quality first; a better model fit to wrong labels learns the wrong thing.
• Retraining vs label quality: retraining on unchanged noisy labels reproduces the ceiling; relabel then retrain.
• Accuracy vs cost vs latency: compare each option by how much accuracy it buys per unit of inference cost (the compute spent each time the model makes a prediction) and added latency (response delay). For example, +1 point of accuracy that doubles serving cost from $2,000 to $4,000 a month is worth it only if that point is worth more than $2,000 a month. Bigger models or heavy features can win accuracy and lose the latency budget.

Flip condition. If the curve were still steep and relabel disagreement under about 5%, collect more labels first.

Direct answer. Decide at each checkpoint from future cost and future expected value only. Money already spent is gone whichever way you choose, so it must not enter the comparison. Set the criteria before the work starts, in writing, with named decision-makers and a rule that stopping is an acceptable outcome. Sunk cost means money, time or effort already spent that cannot be recovered; the sunk cost fallacy is letting it justify more spending.

Set up front (before month one)

• Minimum success thresholds per checkpoint (for example, at month 3 and month 6): a specific measured result that must be reached to continue.
• Kill criteria: conditions that trigger a stop, such as "projected cost to finish exceeds the value ceiling" or "the key assumption is disproved".
• Cost controls: a cap on infrastructure spend per phase, released in tranches, and a named owner who must re-approve each release.
• Who decides: a sponsor plus someone independent of the team, so the team's pride is not the only voice.

At each checkpoint ask three questions

1. What have we learned that changes the odds of success?
2. What is the remaining cost, and what is the payoff if it works?
3. Is there a cheaper path to the same answer: a smaller test, a pivot (a deliberate change of direction using what you learned), or a narrower scope?

Worked example (illustrative, in $ thousands). Spent so far: 1,200 (ignored). Remaining cost to finish: 600. Payoff if it succeeds: 2,400. If the probability of success is 40%, expected value is 0.4 x 2,400 = 960, so net 960 - 600 = +360: continue. If new evidence drops it to 25%, expected value is 0.25 x 2,400 = 600, net 0: break-even, so stop or shrink to a cheaper experiment unless a pivot raises the odds. The 1,200 never appeared in either calculation.

"Encouraging but inconclusive" signals are the dangerous zone: define in advance what would make them conclusive, and cap how long you will wait.

Moving the team when stopping: announce the decision with the evidence, thank the work, name what was learned and reused, give people their next assignments the same week, and write a short retrospective. Stopping cleanly is what makes the next team willing to run an honest test.

Direct answer. Convert every initiative into one common unit (expected annual value in dollars), discount it by confidence, divide by effort, rank, then fill the quarter's capacity from the top and sanity-check the result. Initiatives measured in different units (minutes of delivery time, percentage points of conversion, dollars of inference cost) can only be compared once each is translated into the business outcome it moves.

Method

1. Translate to a common outcome. Delivery time: fewer late orders x repeat-purchase value. Conversion: extra orders x contribution margin. Inference cost: dollars saved directly. Different stakeholders define "impact" differently, so agree the currency with finance and product first.
2. Expected value = value x confidence. Confidence (how likely the estimate holds) is highest for cost savings you can measure directly and lowest for new models with no prior experiment. Give ranges, not a point, and show the range to executives.
3. Divide by effort in person-months to get value per person-month.
4. Fill capacity top-down, then check dependencies, labeling or data limits, and strategic must-dos.

Translating two rows with numbers (illustrative). Delivery time: if the project avoids 20,000 late orders a year and each avoided late order keeps a customer worth $47.50 in repeat purchases, value is 20,000 x 47.5 = $950k. Conversion: 80,000 extra orders a year x $15 contribution margin (revenue minus the variable cost of fulfilling the order) = $1,200k. Fraud model refresh, cold-start recommendations (suggestions for new users or new listings that have no history yet) and generative-AI listing copy are the other three of the six initiatives.

Worked example (illustrative, $k per year).

With 14 person-months of capacity, the top three (3 + 4 + 6 = 13) fit. Conversion is a near-tie with delivery time (75.0 vs 79.2) and has the biggest absolute value, so it is the call to defend: its confidence is the lever. If a quick pilot lifts confidence from 50% to 80%, its score becomes 1,200 x 0.8 / 8 = 120, which moves it to second. Capacity then matters: inference cost (3) + conversion (8) = 11 person-months, and fraud (4) would make 15, over the 14 available, while delivery time (6) would make 17. So the quarter would hold two initiatives plus 3 spare person-months, and you would either find one more person-month for fraud or fund a pilot with the spare.

Handling high uncertainty. For the 30%-confidence bets (cold-start, generative-AI copy), fund a small staged pilot (a few person-weeks to test the riskiest assumption before committing the full build) with a pre-set success bar, rather than a full build. A search-ranking team with a small ML backlog applies the same rule: pilot first, build on evidence. A limited labeling budget (money or time for people to tag training examples) is a constraint to include in effort for any initiative needing new labels.

Presenting to executives. Show the table, the ranges, the one assumption that would change the top three, and what is explicitly not funded. State the opportunity cost of conversion being deferred.

Pitfalls. Comparing raw metrics across units; ranking by absolute value so large, uncertain bets always win; ignoring that effort estimates are the least reliable input.