Feature Success Measurement Questions

Direct answer: Structure the rollback policy around three layers: metric triggers that define when a rollback is even a candidate, severity tiers that determine how fast and how automatically it happens, and a notification plan that scales with severity, so a minor guardrail dip and a major user-harm event are handled by two clearly different paths rather than one blunt rule.

Structured elaboration

1. Metric triggers: define, per feature category, which metrics are rollback-eligible (usually guardrails: error rate, revenue-negative movement, safety/trust signals) and the specific threshold that makes a trigger fire, set before launch, not improvised during an incident.
2. Severity tiers:
   • Tier 1 (automatic, immediate): a metric breaches a bright-line safety or correctness threshold (e.g., error rate above X%, a payment failure spike); rollback is automated, no human approval required, because the cost of delay outweighs the cost of a false-positive rollback.
   • Tier 2 (fast human-in-the-loop): a guardrail metric crosses a concerning but not catastrophic threshold; an on-call owner is paged and must decide within a short window (e.g., 30 minutes) whether to roll back or hold.
   • Tier 3 (scheduled review): a metric is trending in a bad direction but slowly, or the primary metric simply missed its target with no acute harm signal; this goes to the next regular review, not an emergency process.
3. Notification: Tier 1 rollback triggers an automatic incident channel post and a summary to the feature owner and their manager after the fact; Tier 2 pages the on-call owner directly and requires an explicit decision logged; Tier 3 is folded into the normal launch-review cadence with no page.
4. Automated rollback mechanism: for Tier 1, the rollback must be technically instant (a feature flag flip, not a code deploy), since the tier's whole premise is that speed matters more than judgment in the moment.

Worked example: A payments feature has a Tier 1 trigger of "payment failure rate above 2% sustained for 5 minutes" (automatic flag flip, no approval, incident channel auto-posts), a Tier 2 trigger of "refund rate 1.5x baseline sustained for 2 hours" (pages the on-call PM, who must decide within 30 minutes whether to hold or roll back), and a Tier 3 condition of "conversion lift below the pre-agreed threshold at the 30-day mark with no guardrail breach" (goes to the standing weekly launch review, no page).

Trade-offs and pitfalls: A policy with only one tier (either everything is automatic, or everything requires human judgment) is wrong in both directions: fully automatic rollback on soft signals causes noisy, unnecessary reversals that erode trust in the system, while requiring human judgment for a catastrophic, unambiguous safety breach costs precious minutes. The other common gap is defining triggers and tiers but never load-testing the actual rollback mechanism itself, so the "automatic" rollback turns out to take 20 minutes to propagate when the policy assumed it was instant.

Direct answer: Cannibalization in this context means a feature genuinely increases a short-term metric (usually conversion or immediate engagement) by pulling forward or substituting for behavior that would have generated more value later, so the short-term win is partly or entirely offset by a longer-term cost such as lower retention or reduced lifetime value.

Structured elaboration

• The mechanism: a feature that makes an action easier or more appealing right now can shift WHEN users do something (pulling future purchases into the present, which looks like a conversion lift but is not incremental revenue) or WHAT users do (substituting a lower-value action for a higher-value one they would otherwise have taken).
• Why it hides: the short-term metric a launch is judged on is usually measured over a window (days to a few weeks) shorter than the horizon over which the substitution effect plays out (a month or a full purchase cycle), so the launch looks like an unambiguous win before the offsetting cost has had time to show up.
• How to detect it: compare a cohort exposed to the feature against a comparable unexposed cohort over a horizon long enough to capture the behavior the feature might be pulling forward (e.g., if the feature discounts an upcoming purchase, watch purchase frequency for at least one full typical purchase cycle afterward, not just the days right after exposure); a genuine, non-cannibalizing win shows the short-term lift persisting as INCREMENTAL volume over that longer horizon rather than the exposed cohort's later activity dropping below the control cohort's to compensate.
• How to decide whether it is still worth shipping: even a partly cannibalizing feature can be worth keeping if it accelerates revenue recognition, moves users toward a state (subscription, habit) with its own separate value, or if the net effect over the full horizon is still positive once the pulled-forward behavior is accounted for; the decision requires comparing the FULL-HORIZON net effect, not the short-term metric alone.

Worked example: A subscription app adds a prominent "upgrade now, get 20% off this month only" prompt. Short-term: upgrades in month one rise 30% relative to a comparable prior cohort. Full-horizon check: tracking upgrade timing shows a large share of the extra month-one upgrades come from users who, absent the prompt, would have upgraded in month two or three anyway (their subsequent months show no further net-new upgrades relative to control); after accounting for this, incremental full-horizon upgrades are closer to 8%, not 30%, though the discount cost was paid on all of them. The genuine business question then becomes whether the 8% incremental gain and any earlier-revenue-recognition value outweigh the discount cost.

Trade-offs and pitfalls: The most common mistake is stopping the analysis at the short-term metric because it is the one the launch was scored on, and never running the longer-horizon comparison that would reveal cannibalization. The opposite mistake is assuming every short-term win must be cannibalization and discounting all fast results, which under-credits features whose value genuinely materializes quickly and holds.

Direct answer: Compute the net daily revenue by multiplying visitors by conversion rate by average order value in each state and comparing the two totals; here, the feature is net positive, generating an estimated $880,000 more per day, because the relative gain in conversion rate (20%) outweighs the relative drop in average order value (2%).

Structured elaboration

• Revenue per day is the product of three quantities: total visitors, conversion rate, and average order value (AOV). Since visitors are held constant here, comparing before and after isolates the combined effect of the conversion-rate change and the AOV change.
• $\text{Revenue}_{\text{before}} = 1{,}000{,}000 \times 0.10 \times \$50 = \$5{,}000{,}000$
• $\text{Revenue}_{\text{after}} = 1{,}000{,}000 \times 0.12 \times \$49 = \$5{,}880{,}000$
• $\Delta \text{Revenue} = \$5{,}880{,}000 - \$5{,}000{,}000 = \$880{,}000 \text{ per day (+17.6\%)}$
• The intuition behind why a smaller-looking AOV drop (2%) is dominated by a larger-looking conversion gain (20% relative) is that the conversion improvement applies to the ENTIRE new set of purchasers created by the higher rate, while the AOV drop applies only to the (larger) pool of orders now happening; multiplying it out is what resolves the apparent tension rather than eyeballing the two percentages against each other.
• Before declaring this a clean net-positive result, state the assumptions: this assumes the conversion-rate and AOV changes are both statistically real (not noise) and causally attributable to the feature (not, say, a concurrent promotion), and that the AOV drop will not compound over time (e.g., if the feature specifically nudges people toward cheaper items, watch whether AOV keeps sliding in subsequent weeks rather than stabilizing at the new $49 level).

Worked example: If the AOV drop had instead been to $41 (an 18% relative drop) rather than $49, the same calculation would flip the sign: $1{,}000{,}000 \times 0.12 \times $41 = $4{,}920{,}000$, an $80,000 daily LOSS versus baseline, despite the same 20% relative conversion gain. This shows why the "is it net positive" question cannot be answered by comparing the two percentage changes directly; the actual dollar multiplication is required, and a seemingly large conversion win can still be a net loss if the AOV drop is large enough.

Trade-offs and pitfalls: The most common mistake is eyeballing "conversion up more than AOV down, so it must be a net win" without doing the multiplication, which happens to work in this exact example but is not reliably true in general, as the alternate scenario above shows. The other pitfall is treating the point-in-time $880,000 figure as a permanent daily run-rate without checking whether the underlying rates are stable or still shifting (a common issue if the feature is new and still ramping or subject to a novelty effect).

Direct answer: Feature success measures whether a shipped feature achieved its own local goal (adoption, usage, satisfaction for that feature). Product success measures whether the product as a whole is winning (retention, revenue, market position). A feature can succeed locally and still fail to matter at the product level if its goal is not actually causally linked to a product-level outcome, or if its effect is too small to show up against everything else moving the product metric.

Structured elaboration

• Feature-level metrics are local and fast-moving: click-through on a new button, completion rate of a new flow, adoption of a new setting. They tell you whether people used and liked the thing you built.
• Product-level metrics are the north-star and its supporting metrics: overall retention, revenue per user, weekly active users. They tell you whether the business is healthier.
• The gap between them opens for two structural reasons: (1) the feature's local metric is not on the causal path to any product metric (you built something people like but it does not change behavior that matters to the business), or (2) the feature is on the causal path but its effect size is too small relative to the product metric's overall variance and other drivers to be detectable.
• A senior candidate treats a high-adoption, no-product-impact result as informative, not as a failure to hide: it tells you the local metric was mis-specified as a proxy for value, or that the feature needs to be paired with something else to convert usage into value.

Worked example: A photo-sharing app ships a new sticker pack for its Stories feature. Feature-level metric: 40% of daily Stories creators use a sticker at least once (strong adoption). Product-level metric: overall app retention is flat. Investigation shows sticker usage is concentrated among users who were already highly engaged and would have stayed regardless; the feature added a small delight moment but did not reach or change behavior for users at risk of churning, who are the population retention actually needs to move. The feature is a real, well-adopted feature success and a real product-success non-event, and both statements are true at once.

Deciding whether the feature is still worth keeping: A high-adoption, no-product-impact result does not by itself answer the keep-or-remove question; work through four checks before deciding. First, rule out a measurement problem: confirm the evaluation window was long enough, and that the product metric was even plausibly capable of moving by an amount this feature's expected effect size could produce. Second, check whether the feature's adopters overlap with the population the product metric actually needs to move (in the sticker-pack example, adopters who were already retained tell you little about churn-risk users, so the feature may simply have never been positioned to move retention). Third, price the ongoing cost of keeping the feature (engineering maintenance, support burden, added product complexity) against the cost of removing it (backlash from an adopted user base, loss of a competitive differentiator, loss of any defensive or long-horizon value not yet visible in the metric). Fourth, weigh those two: if the feature is cheap to maintain and removing it risks disappointing a real, adopted user base for no measurable product gain, keep it as a low-cost, retention-neutral feature; if it is expensive to maintain, crowds out higher-impact work, or its adopters are not a population the business actually needs to serve, sunset it rather than keeping it on the faith that it will eventually pay off.

Trade-offs and pitfalls: The common mistake is to treat "no product-metric movement" as proof the feature failed, which incentivizes teams to pick feature metrics that are easy to move (vanity engagement) rather than metrics genuinely on the path to product value. The opposite mistake is under-crediting a feature whose real payoff is defensive (it kept churn-risk users from leaving) or long-horizon (it builds a habit that pays off in a later quarter), where the effect will not show in a same-quarter product metric at all.

Direct answer: Separate the decision from the sunk cost explicitly, using the data as the anchor: state clearly that the campaign investment already spent is gone either way and cannot be recovered by continuing to run a feature that is now shown to be harmful, then focus the conversation on the go-forward cost of keeping versus rolling back.

Structured elaboration

• Name the sunk-cost reasoning directly and respectfully, since stakeholders pushing back are often doing so in good faith, not irrationally; acknowledge the investment was real and the intention behind it was reasonable, while being clear that the investment's size has no bearing on what happens going forward.
• Reframe the decision around the go-forward numbers only: what does keeping the feature cost from today onward (continued harm shown in the data), versus what does rolling back cost from today onward (any remaining campaign value that would be lost, any embarrassment or process cost of reversing course); compare those two forward-looking costs, not the money already spent.
• Bring the data itself into the room as the anchor for the conversation, rather than relying on personal authority to win the argument; if possible, quantify the ongoing cost of keeping the feature live in the same terms stakeholders already care about (the campaign's ROI, updated to reflect the newly-discovered harm) so the conversation happens on shared ground.
• If stakeholders still resist, propose a middle path that respects both the data and the organizational reality: a partial rollback, a time-boxed fix-and-recheck window, or a scoped rollback to the specific segment where the harm concentrates, rather than an all-or-nothing framing that invites maximal resistance.

Worked example: A promotional feature tied to a $2M marketing campaign shows clear experiment data that it reduces long-term customer value for a meaningful segment of exposed users. Engineering and marketing resist a full rollback, citing the campaign investment. The response separates the two questions explicitly: "the $2M is spent regardless of what we do today; the question in front of us is whether continuing to run this feature costs us more in ongoing customer value than the marginal campaign benefit we would lose by stopping now." Presenting the ongoing cost in the same dollar terms as the original campaign's expected ROI reframes the conversation productively, and the team agrees to a scoped rollback limited to the segment where the harm is concentrated, preserving the campaign's benefit for the segment where no harm was detected.

Trade-offs and pitfalls: The most common mistake is treating this purely as a persuasion problem (finding the right words to win the argument) rather than a framing problem (getting everyone to evaluate the same, correctly-scoped forward-looking comparison); once the framing is right, the argument often resolves itself. The other pitfall is proposing an all-or-nothing rollback when a scoped, partial option would address the actual harm while preserving legitimate value elsewhere, which needlessly maximizes organizational resistance to a decision that did not need to be all-or-nothing.