Feature Success Measurement Questions

Direct answer: Before answering "has it succeeded," ask what success was supposed to look like: what was the feature's stated goal, what single metric was chosen to represent that goal, what threshold counts as a win, and over what time window. Without those four answers pinned down, "succeeded or not" has no stable meaning.

Structured elaboration

1. Scope the goal: ask what problem the feature was meant to solve and for whom, since a feature can be judged only against the goal it was actually built for, not an ambient hope that it would help everything.
2. Confirm the metric: ask what the team originally proposed as the primary metric, and check that it is actually causally close to the stated goal rather than a convenient proxy (adoption is not the same thing as value).
3. Confirm the threshold and baseline: ask what number would have counted as success versus failure, and what the pre-launch baseline was, since "it went up" is meaningless without a baseline and a bar.
4. Confirm the time window and confounders: ask how long the feature has been live and whether anything else changed in that window (a pricing change, a seasonal effect, another feature launch) that could explain any movement independent of this feature.
5. Check guardrails: ask whether anything the feature was NOT supposed to affect moved in a bad direction, since a metric win with a hidden guardrail loss is not a clean success.
6. Only once these are answered does "has it succeeded" become answerable, and often the honest first answer is "we do not have enough of the above defined yet to say."

Worked example: A PM says "we launched personalized push notifications, did it succeed?" Structured questions surface that the original goal was reducing app-open latency for lapsed users, the chosen metric was 7-day reactivation rate, the pre-launch baseline was 4%, the target was 6%, the feature has been live three weeks (short of the planned six-week evaluation window), and a separate marketing campaign ran during the same window. The honest answer is not yet: reactivation is currently at 5.5% but it is too early relative to the evaluation window and confounded by the campaign, so the correct next step is to wait for the full window and, if possible, exclude or control for the campaign-exposed cohort.

Trade-offs and pitfalls: The common failure is answering the question with whatever metric moved most favorably in whatever time window happens to be available, which produces a flattering but unreliable answer. The opposite failure is refusing to give any read at all and hiding behind "we need more data" indefinitely; a senior candidate gives the best available honest read (including "too early to tell, here is the leading indicator so far") rather than either extreme.

Direct answer: In the first 30-90 days after a major launch, track six categories of signal, one metric each: product adoption, user engagement depth, revenue impact, support/complaint volume, technical performance, and a guardrail metric specific to the feature's biggest plausible failure mode. For each, set a target, a detection method, a likely root cause if it is missed, and a remediation to try.

Structured elaboration

1. Adoption (% of eligible users who tried the feature at least once). Target: a stated reach goal (e.g., 25% of eligible users try it within 60 days). Detect: compare the actual adoption curve against the target curve weekly; tracking meaningfully below it by day 30 is the signal. Root cause: usually a discoverability problem, the feature is not surfaced where users would notice it, or its value proposition is unclear from the entry point. Remediation: a discoverability fix (onboarding prompt, better placement, in-product nudge), not a rollback, since low adoption alone does not mean the feature is bad.
2. Engagement depth (repeat usage rate among adopters). Target: a stated repeat-usage rate (e.g., 40% of adopters use it again within two weeks). Detect: compare repeat-usage rate among the adopter cohort against that target. Root cause: typically a specific friction or drop-off point in the flow itself, adopters try it once but hit a confusing step, an unclear payoff, or a bug that discourages a second use. Remediation: instrument and review the specific drop-off step, then fix that friction point directly (simplify the step, clarify the value shown after first use).
3. Revenue (attributable revenue per exposed user). Target: tied to the business case that justified building the feature (e.g., $X per exposed user within 60 days). Detect: compare attributable revenue per exposed user against target, controlling for cohort size and exposure window. Root cause: either the adoption-times-engagement mechanism is working but not yet monetizing (too early, or missing a monetization hook), or the underlying mechanism itself is not reaching users who would pay. Remediation: if the mechanism works but is not monetizing, add or tune a monetization hook (an upsell prompt, a pricing tier); if the mechanism itself is broken, redesign the core flow rather than tuning monetization.
4. Support/complaint volume (tickets mentioning the feature per 1,000 exposed users). Target: a ceiling (e.g., under 2 tickets per 1,000 exposed users). Detect: tag and count feature-specific tickets weekly against that ceiling. Root cause: usually a specific UX confusion point, complaint volume concentrates in a narrow theme (a mislabeled control, an unclear state) rather than spreading evenly. Remediation: triage the top complaint theme and ship a targeted UX fix (copy change, added confirmation step) rather than a full rollback.
5. Technical performance (feature-specific error rate or latency). Target: a ceiling (e.g., p95 latency under 500ms, error rate under 0.5%). Detect: monitor the feature-specific error rate and latency percentile continuously against the ceiling. Root cause: commonly an unoptimized code path, an under-provisioned dependency, or a new failure mode the feature introduced (a new external call, a new data write path). Remediation: an engineering fix targeted at the specific failure point, and if the breach is severe, a partial rollback while the fix ships.
6. Feature-specific guardrail (chosen per feature; e.g., for a recommendation feature, diversity of recommended items; for a pricing feature, refund rate). Target: set from the single worst plausible failure mode for this specific feature (e.g., refund rate should not exceed its pre-launch baseline by more than 1 relative point). Detect: monitor the chosen guardrail metric against that feature-specific threshold. Root cause: depends on the mechanism at risk, for a pricing feature a refund spike often traces to a specific price point or misleading messaging that oversold the offer. Remediation: address the specific mechanism identified (adjust the price point or messaging), not a generic response, since a bespoke guardrail requires a bespoke fix tied to its specific failure mode.

Worked example: A ride-sharing app launches in-app trip-sharing with a contact. Targets: adoption 15% of trips within 60 days; engagement 40% of adopters use it again within a month; revenue neutral (feature is a trust/safety play, not a monetization one); support tickets under 2 per 1,000 exposed trips; p95 share-link generation latency under 500ms; guardrail is that under 1% of shares are sent to a number never contacted before (a proxy against misuse). At day 60: adoption is 9% (missed), engagement among adopters is 55% (exceeded), support tickets are 1.1 per 1,000 (fine), latency is 480ms (fine), misuse guardrail is 0.4% (fine). Verdict: the mechanism works well once discovered, so the remediation is a discoverability push (in-trip prompt), not a redesign or rollback.

Trade-offs and pitfalls: Picking six generic metrics without a feature-specific guardrail misses the failure mode most likely to actually hurt users or the business for that particular feature; the guardrail should be argued from "what is the worst plausible way this specific feature goes wrong," not copied from a template. The other pitfall is setting remediation only for a metric miss and not for a metric that is suspiciously and uniformly too good (which often signals an instrumentation bug rather than a true win).

Direct answer: Do not let any single metric decide the verdict; build a small weighted scoring model that expresses how much each signal matters in dollars or dollar-equivalents, then read the sign and size of the combined score, while treating a large enough move in any one signal as a veto regardless of the composite.

Structured elaboration

• Convert each signal to a comparable unit where possible. Weekly active users (WAU) growth converts to expected revenue via the product's revenue-per-active-user; support tickets convert to a cost via the fully-loaded cost per ticket; Net Promoter Score (NPS) is harder to convert directly to dollars, so treat it as a qualitative veto input rather than forcing a dollar figure onto it.
• Build a simple weighted formula for the two dollar-convertible signals, and treat the NPS movement as a separate check: if NPS drops beyond an agreed threshold, that overrides a positive dollar score, because a trust erosion compounds in ways the current quarter's revenue math will not capture.
• State the formula and its inputs explicitly so the calculation is reproducible, and recompute it whenever any input assumption (revenue-per-user, cost-per-ticket) changes materially.

Worked example: Assume this product's average revenue per active user is $2/week, and each support ticket costs the company $8 fully loaded. A 4% increase in WAU on a base of 500,000 weekly actives is 20,000 incremental actives:

$$\text{Revenue gain} = 20{,}000 \times \$2 = \$40{,}000 \text{ per week}$$

A 6% increase in support tickets on a base of 3,000 tickets/week is 180 additional tickets:

$$\text{Support cost} = 180 \times \$8 = \$1{,}440 \text{ per week}$$

The net dollar score is:

$$\$40{,}000 - \$1{,}440 = \$38{,}560 \text{ per week, strongly positive}$$

That leaves the NPS drop of 0.8 points as the deciding factor: if the team's pre-agreed NPS veto threshold is a drop of more than 1.0 point, 0.8 does not trigger the veto, and the verdict is keep, with a note to monitor NPS for a further slide. If the threshold were 0.5 points, the verdict would flip to iterate despite the strongly positive dollar score, because the veto is designed to catch exactly this situation.

Trade-offs and pitfalls: Forcing every signal into a single dollar-weighted number is tempting but dangerous when one signal (like NPS) measures something the model cannot price accurately; treating it as a hard veto rather than folding it into the weighted average avoids letting a large, easy-to-compute financial number silently outvote a smaller but more consequential relationship signal. The other pitfall is picking the revenue-per-user and cost-per-ticket conversion factors without stating them, which makes the "answer" look objective while actually hiding a set of assumptions a reviewer cannot check.

Direct answer: Cannibalization in this context means a feature genuinely increases a short-term metric (usually conversion or immediate engagement) by pulling forward or substituting for behavior that would have generated more value later, so the short-term win is partly or entirely offset by a longer-term cost such as lower retention or reduced lifetime value.

Structured elaboration

• The mechanism: a feature that makes an action easier or more appealing right now can shift WHEN users do something (pulling future purchases into the present, which looks like a conversion lift but is not incremental revenue) or WHAT users do (substituting a lower-value action for a higher-value one they would otherwise have taken).
• Why it hides: the short-term metric a launch is judged on is usually measured over a window (days to a few weeks) shorter than the horizon over which the substitution effect plays out (a month or a full purchase cycle), so the launch looks like an unambiguous win before the offsetting cost has had time to show up.
• How to detect it: compare a cohort exposed to the feature against a comparable unexposed cohort over a horizon long enough to capture the behavior the feature might be pulling forward (e.g., if the feature discounts an upcoming purchase, watch purchase frequency for at least one full typical purchase cycle afterward, not just the days right after exposure); a genuine, non-cannibalizing win shows the short-term lift persisting as INCREMENTAL volume over that longer horizon rather than the exposed cohort's later activity dropping below the control cohort's to compensate.
• How to decide whether it is still worth shipping: even a partly cannibalizing feature can be worth keeping if it accelerates revenue recognition, moves users toward a state (subscription, habit) with its own separate value, or if the net effect over the full horizon is still positive once the pulled-forward behavior is accounted for; the decision requires comparing the FULL-HORIZON net effect, not the short-term metric alone.

Worked example: A subscription app adds a prominent "upgrade now, get 20% off this month only" prompt. Short-term: upgrades in month one rise 30% relative to a comparable prior cohort. Full-horizon check: tracking upgrade timing shows a large share of the extra month-one upgrades come from users who, absent the prompt, would have upgraded in month two or three anyway (their subsequent months show no further net-new upgrades relative to control); after accounting for this, incremental full-horizon upgrades are closer to 8%, not 30%, though the discount cost was paid on all of them. The genuine business question then becomes whether the 8% incremental gain and any earlier-revenue-recognition value outweigh the discount cost.

Trade-offs and pitfalls: The most common mistake is stopping the analysis at the short-term metric because it is the one the launch was scored on, and never running the longer-horizon comparison that would reveal cannibalization. The opposite mistake is assuming every short-term win must be cannibalization and discounting all fast results, which under-credits features whose value genuinely materializes quickly and holds.

Direct answer: Build the rubric around the primary metric first, then let the other two metrics adjust the verdict rather than override it outright: a primary-metric win with two supporting signals in line is a success; a primary-metric win with one supporting signal moving the wrong way is an iterate; a primary-metric miss, or a severe move in Net Promoter Score or revenue per user, is a rollback.

Structured elaboration

• Rank the three metrics by what they protect: conversion rate (CVR) is the primary evidence the feature does what it was built to do; revenue per user (RPU) is the business-health guardrail; Net Promoter Score (NPS) is the long-term-relationship guardrail. Treat CVR as necessary but not sufficient.
• Set directional bands rather than a single pass/fail line: CVR up beyond a pre-agreed minimum detectable lift is required to even consider "success." Within that, classify by the guardrails:
  • CVR up, RPU flat or up, NPS flat or up -> success, keep as-is.
  • CVR up, one guardrail down but within a small pre-agreed tolerance (e.g., NPS down less than half a point) -> iterate, ship but fix the specific friction driving the guardrail dip.
  • CVR up, a guardrail down beyond tolerance, or CVR flat/down regardless of guardrails -> rollback.
• Make the tolerance bands explicit and numeric before launch, not vibes-based after the fact; write them into the same launch plan that defines the primary metric.
• Treat a guardrail breach as an asymmetric veto: a large enough NPS or RPU drop overrides a CVR win, because the rubric exists specifically to prevent a locally-positive feature from harming the relationship or the business model.

Worked example: Suppose the pre-agreed bands are: CVR must be up by at least 2 relative percentage points to count; NPS tolerance is a drop of up to 0.3 points; RPU tolerance is a drop of up to 1%. Observed result: CVR +3.5% relative, NPS -0.5 points, RPU +0.8%. CVR clears its bar, RPU is fine, but NPS breached its 0.3-point tolerance. Under the asymmetric-veto rule this is an iterate, not a straight success: ship the underlying mechanism (it clearly works) but treat the NPS dip as a signal to investigate what specifically is annoying users (a common cause is a feature that increases short-term conversion via added friction elsewhere, like a more aggressive upsell prompt) before declaring full success.

Trade-offs and pitfalls: A rubric with only additive scoring (sum the three metrics with weights) hides exactly the asymmetric-risk case above: a good CVR score can numerically outweigh a bad NPS score even though the two are not equally reversible (a users' trust drop can compound in ways a conversion-rate gain cannot offset). The other common pitfall is setting the tolerance bands loosely enough that every result becomes a "success," which defeats the point of having a rubric at all; the bands should be tight enough that rollback and iterate verdicts genuinely occur some fraction of the time.