Metrics and KPI Design Questions

Direct answer

Absolute churn rate (also called customer or logo churn) is the percentage of customers lost in a period; revenue churn rate is the percentage of recurring revenue lost in that period, usually measured against starting MRR (monthly recurring revenue). They diverge whenever the customers who leave are not representative of the average customer: losing a few high-paying accounts can shrink revenue churn even when the customer count looks fine, and vice versa. A team that tracks only customer churn can miss a revenue problem building up in its largest accounts.

Structured elaboration

Definitions

• Customer churn rate = customers lost in period / customers at start of period.
• Revenue churn rate = MRR lost in period (from cancellations and downgrades) / MRR at start of period.

Because customer churn counts every account equally, it is blind to price. Revenue churn weights each departure by its dollar size, so it reacts to who left, not just how many. The gap between the two metrics is itself a diagnostic: rising revenue churn alongside flat or falling customer churn usually points at concentration risk in high-value accounts, expansion not offsetting churn among large accounts, or a pricing and segmentation shift underway.

Worked example

Start of Month 1: 100 customers. 20 large accounts at $2,000/month = $40,000 MRR, and 80 small accounts at $750/month = $60,000 MRR. Total starting MRR = $100,000 (20 x $2,000 + 80 x $750 = $40,000 + $60,000).

Month 1: 8 small accounts cancel.
$\text{Customer churn}_1 = \frac{8}{100} = 8\%$
$\text{Revenue churn}_1 = \frac{8 \times \$750}{\$100{,}000} = \frac{\$6{,}000}{\$100{,}000} = 6\%$

Month 2 (same starting base: 100 customers, $100,000 MRR): only 4 large accounts cancel, no small-account churn.
$\text{Customer churn}_2 = \frac{4}{100} = 4\%$
$\text{Revenue churn}_2 = \frac{4 \times \$2{,}000}{\$100{,}000} = \frac{\$8{,}000}{\$100{,}000} = 8\%$

Result: customer churn fell (8% to 4%) while revenue churn rose (6% to 8%), because churn shifted from small, low-ARPU (average revenue per account) accounts to large, high-ARPU accounts.

Interpretation: fewer customers left overall, but the ones who left carried disproportionate revenue weight. This points at concentration risk in the largest accounts and possibly a pricing or segmentation issue, and argues for splitting churn dashboards by ARPU band or account tier rather than reporting one blended number.

Trade-offs & pitfalls

• Reporting only customer churn understates real financial exposure when the base has a long tail of small accounts and a few large ones.
• Reporting only revenue churn can look artificially healthy if a company over-serves a few large accounts while quietly losing many small customers, masking a broader satisfaction problem.
• A common pitfall is computing revenue churn against ending MRR instead of starting MRR, or mixing expansion revenue into the same ratio; keep churn and expansion as separate line items so this ratio measures loss only.
• Net revenue churn (which nets expansion/upsell against contraction and cancellations) is a different, usually more favorable, signal than gross revenue churn; naming the exact variant matters when comparing figures across teams.

Direct answer

Treat "engagement" as a symptom, not a metric: run a short discovery to find the actual decision the stakeholder needs to make, translate that decision into two or three concrete metrics with explicit formulas, and validate each one by showing the stakeholder what action it would trigger before it ever ships to a dashboard. The quantified version of this request ("grow engagement 10%") makes the ambiguity worse, not better, until the target itself is pinned down.

Structured elaboration

Step 1: Clarify the decision, not the word. Ask what the stakeholder would do differently if the number went up versus down, over what time horizon, for which user segment, and at what granularity. "Engagement" said by a growth lead usually means retention risk; said by a content team it usually means session value; said by a monetization team it often means feature adoption tied to revenue. The word alone does not tell you which.

Step 2: Translate to metrics with explicit formulas.

• Active rate: daily active users over monthly active users (DAU/MAU), computed as distinct users in a 24-hour window divided by distinct users in the trailing 30-day window, by cohort and platform. Signals habitual use; a low value points to retention work.
• Median session value: session duration and sessions per user per week. Signals whether the product is pulling people back in; a falling trend points to a content or UX investigation.
• Feature conversion funnel: view to click to complete for a target feature, with drop-off percentage at each step. Points directly at where to intervene: messaging, UX, or the feature itself.

Step 3: Validate against the ambiguous target itself. If the ask came with a number ("grow engagement 10%"), that number is often as underspecified as the word. Before building anything, confirm what "10%" is relative to, because a relative-growth reading and an absolute-percentage-point reading of the same instruction produce very different targets (worked below), and shipping the wrong one wastes a quarter.

Step 4: Prototype and walk through with the stakeholder. Build a mock dashboard with real sample data and ask: "if this metric moved from X to Y, what would you do?" If the stakeholder cannot answer, the metric is not actionable yet and needs another iteration. Capture a threshold, an owner, and a review cadence for each metric before calling the translation done.

Worked example

Suppose the baseline active rate (DAU/MAU) is 0.35, and the stakeholder asks for "10% more engagement" without saying what that 10% is measured against. Two honest readings of the same instruction:

$\text{Relative reading: } 0.35 \times 1.10 = 0.385$
$\text{Absolute reading: } 0.35 + 0.10 = 0.45$

The relative reading targets an active rate of 38.5%, a modest lift. The absolute reading targets 45%, which is a 10-percentage-point jump, nearly 29% relative growth over the same baseline:

$\frac{0.45 - 0.35}{0.35} \approx 0.286$

Those two targets imply very different roadmaps: the first is achievable with incremental funnel fixes, the second likely requires a structural change to the product's habit loop. This is exactly why step 3 above exists: the quantified ask does not remove the ambiguity, it just moves it one level deeper, into what the percentage is a percentage of.

Trade-offs & pitfalls

The main failure mode is skipping straight to a metric because it is easy to compute (total sessions, total time-in-app) rather than one that maps to a decision; that produces a dashboard nobody acts on. A second failure mode is over-fitting the translation to whichever stakeholder is loudest, producing a metric set that serves one team's decision but not the actual business question. Watch for gameable proxies too: session duration can be inflated by confusing navigation, and a feature-funnel completion rate can be inflated by making the "complete" step trivially easy. The fix in both cases is pairing the primary metric with a guardrail (task success rate, support-ticket rate) so an intervention that juices the number without creating real value gets caught.

Direct answer

For a daily active users ("DAU") dashboard, choose segments that map to the two questions a product manager actually asks when DAU moves: is this an acquisition-or-retention story, or a platform-or-technical story. Expose enough of them that one view can localize the cause, rather than just confirming that something moved.

Structured elaboration

Worked example

Today's DAU is 50,000, of which 12,000 are new users and 38,000 are returning:

Yesterday's DAU was 48,000, with an 18% new-user share (8,640 new, 39,360 returning). Headline DAU growth:

looks like modest, healthy growth. But the returning-user count actually fell:

All of the net DAU growth came from a spike in new users, likely a campaign, while the returning-user base, the harder metric to move and the one that indicates real product stickiness, shrank by about 3.5%. The aggregate DAU number alone hides this; the new-vs-returning segment is what surfaces it.

Trade-offs and pitfalls

Putting too many segments on one dashboard turns it into noise; prioritize the two or three most decision-relevant ones for an executive view and push the rest to a drill-down. Small segments, a minor platform or a small region, can show large percentage swings purely from small sample sizes, so alerts on those segments need a minimum-volume threshold before firing. Segments can also interact (a channel's users may cluster heavily on one platform), so a single-dimension view can still mislead if the combination isn't checked. Finally, segmentation only localizes a change; it doesn't by itself explain the cause, so it should be the first step of an investigation, not the conclusion of one.

Direct answer

Build a hypothesis tree with four branches (user mix, pricing/discounting, product mix, measurement change) and use a mix-versus-within-segment decomposition to quantify how much of the revenue-per-user change each branch actually explains, rather than guessing from a dashboard. Conversions rising while revenue per user falls is the classic signature of a lower-value segment or a discounted product line growing its share of the base.

Structured elaboration

Before trusting any of these, confirm the conversion metric's definition did not change over the same window; a widened definition of "conversion" (for example, counting a free trial start) can produce this exact pattern on its own with no real change in buyer behavior.

The core quantitative tool is a mix-versus-within-segment decomposition. Revenue per user, averaged across segments, is a weighted average of each segment's own revenue per user, weighted by that segment's share of total users. The change in the overall average can always be split exactly into two parts: how much moved because the segment weights shifted (mix effect), holding each segment's own value fixed, and how much moved because a segment's own value changed (within-segment effect), holding the new weights fixed:

$\Delta RPU = \underbrace{\sum_i (w_i^{after} - w_i^{before}) \cdot RPU_i^{before}}_{\text{mix effect}} \;+\; \underbrace{\sum_i w_i^{after} \cdot (RPU_i^{after} - RPU_i^{before})}_{\text{within-segment effect}}$

This identity holds exactly for any two periods; it is algebra, not a statistical estimate, so it always reconciles to the observed total change.

Worked example

Two segments: Segment A (existing, higher-value customers) and Segment B (newly acquired, promo-driven, lower-value customers).

Before: A is 80% of converters at $60 revenue per user; B is 20% at $30.

$\overline{RPU}_{before} = 0.80(60) + 0.20(30) = 48 + 6 = 54$

After: total converters rose (consistent with "conversions increased"), but the mix shifted to 70% A / 30% B, and Segment A's own revenue per user also fell to $55 due to a new discount, while Segment B's stayed at $30.

$\overline{RPU}_{after} = 0.70(55) + 0.30(30) = 38.5 + 9 = 47.5$

$\Delta RPU = 47.5 - 54 = -6.5$

Decomposing that -6.5 change:

$\text{Mix effect} = (0.70 - 0.80)(60) + (0.30 - 0.20)(30) = -6 + 3 = -3.0$
$\text{Within-segment effect} = 0.70(55-60) + 0.30(30-30) = -3.5 + 0 = -3.5$
$-3.0 + (-3.5) = -6.5 \quad \checkmark$

So of the $6.50 drop in revenue per user, about 46% (-3.0) is explained by the shift toward the lower-value segment, and about 54% (-3.5) is explained by the discount applied within the existing segment. That total revenue may still be flat or up (since total converters grew) does not contradict this; the decomposition explains the per-user average, and total revenue is a separate question the same segment data can answer once volume is added back in.

Trade-offs & pitfalls

The decomposition assumes segments are the right unit of analysis; if mix and pricing move together for a real underlying reason (a new discount was specifically targeted at the segment growing fastest), the two effects are correlated and the split, while arithmetically exact, is less causally clean than it looks. A second pitfall is Simpson's-paradox-style masking: if segments are defined too coarsely, mix and within-segment effects computed at that coarse level can hide a sharper story visible only in a finer segmentation. A third is treating "product changes" as a residual explanation once the others are ruled out, without direct evidence; that is a hypothesis of last resort, not a conclusion. Finally, always check the measurement-change branch first: it is the cheapest to confirm or rule out, and skipping it risks running an expensive segment investigation to explain an artifact.

Direct answer

Split the decision into two layers: a guardrail that can veto a launch outright regardless of overall score, and a weighted composite of the primary and secondary metric for cases that clear the guardrail. A guardrail breach (for example, retention dropping further than the pre-agreed acceptable floor) should reject or delay the launch even when the primary metric looks great, because the guardrail exists precisely to catch the case where a big headline win is masking real long-term damage.

Structured elaboration

Metric roles, set before the experiment runs

• Primary (the objective-aligned metric the experiment is testing for): weight roughly 0.7.
• Secondary (a health metric that should not be sacrificed for the primary): weight roughly 0.3, plus a hard guardrail floor.
• Guardrail: a maximum acceptable move on the secondary metric, agreed before launch, that vetoes the decision regardless of the weighted score.

Decision tree

graph TD
  A[Primary metric statistically and practically significant?] -->|No| B[Iterate or extend the test]
  A -->|Yes| C[Does any guardrail breach its veto threshold?]
  C -->|Yes| D[Reject / iterate, regardless of score]
  C -->|No| E[Compute weighted score]
  E -->|Score above +0.2| F[Approve launch]
  E -->|Score between -0.2 and +0.2| G[Targeted follow-up: qualitative research, cohort analysis]
  E -->|Score below -0.2| H[Reject]

Incorporating qualitative signals

Treat qualitative severity as its own veto tier: a critical qualitative signal (the product is unusable for a segment, a spike in complaint volume) should route straight to the reject/iterate branch even if it arrives after the quantitative score already looked acceptable, the same way a quantitative guardrail breach does.

Worked example

Given the scenario in the question, revenue is up 5% and retention is down 3%. Set the guardrail at a maximum acceptable absolute retention drop of 2%, and normalize percentage moves on a scale where a 10-point move equals a full unit of business impact:

The weighted score of 0.26 is above the +0.2 approval line, which would normally mean "approve." But the retention drop of 3% exceeds the 2% guardrail floor:

The guardrail check runs before the weighted score is even consulted, so the council rejects (or requires a mitigation plan and a short staged pilot) despite the positive weighted score. This is the entire point of a two-layer decision: a single composite number would have said "launch," and the guardrail is what catches the case the composite alone would have missed.

Trade-offs & pitfalls

A weighted composite score is easy to compute but easy to game by tuning the weights after seeing the result; weights and guardrail floors have to be pre-registered before the experiment reads out, not chosen once the numbers are in. Guardrails that are too tight reject changes that would have been fine long-term (a temporary retention dip that recovers), while guardrails set too loose let real long-term damage through under revenue cover; recalibrate the guardrail periodically against actual outcomes, not just intuition. A common wrong turn is letting a strong primary-metric win talk the council out of a guardrail breach in the room ("but revenue is up so much"); the guardrail only works as a hard veto if it is treated as non-negotiable at decision time, with any exception requiring an explicit, documented override rather than an implicit one.