Metrics and KPI Design Questions

Direct answer

Both KPIs (key performance indicators) need a baseline computed from enough history to separate a real problem from normal seasonality, a warning threshold that gets a look from the team, and a critical threshold that pages someone immediately with a concrete first action. DAU (daily active users) uses a same-weekday rolling baseline because weekly seasonality dominates it; checkout conversion uses a rolling baseline plus a statistical confidence interval, because with a large enough sample size, even small real drops are detectable well before they would cross a big round-number threshold.

Structured elaboration

General practice: suppress alerts during known experiments or marketing campaigns by annotating the baseline calculation with those dates, and require a second, unrelated signal, such as a spike in server error rate, before treating a metric drop as critical rather than an anomaly worth a quick look.

Worked example

DAU: same-weekday 28-day rolling median = 50,000.
$\text{Warning threshold} = 50{,}000 \times (1 - 0.15) = 42{,}500$
$\text{Critical threshold} = 50{,}000 \times (1 - 0.30) = 35{,}000$
If today's actual DAU is 40,000:
$\frac{50{,}000 - 40{,}000}{50{,}000} = 20\%\text{ drop}$
20% is past the warning line (15%) but below the critical line (30%), so this fires a warning, not a page.

Checkout conversion: baseline conversion rate $p = 3.0\%$ on a typical day of $n = 10{,}000$ checkout starts. Standard error of a daily rate at this baseline:
$SE = \sqrt{\frac{p(1-p)}{n}} = \sqrt{\frac{0.03 \times 0.97}{10{,}000}} = \sqrt{0.00000291} \approx 0.17\%$
A 95% confidence interval around the baseline:
$3.0\% \pm 1.96 \times 0.17\% \approx [2.67\%,\ 3.33\%]$
If today's observed conversion is 2.5% (250 of 10,000):
$\frac{3.0\% - 2.5\%}{3.0\%} \approx 16.7\%\text{ relative drop}$
2.5% is below the lower confidence bound (2.67%) and the relative drop (16.7%) exceeds the 10% warning line but not the 25% critical line, so this also fires a warning: real enough to be outside normal daily noise, but not yet severe enough to page.

Trade-offs & pitfalls

• A same-weekday baseline needs enough history to be stable; a newly launched product or a recent step-change in traffic, such as a marketing campaign or a platform migration, makes the rolling median itself wrong until enough post-change data accumulates.
• The confidence-interval approach for checkout conversion assumes a roughly constant daily sample size; a big swing in checkout-start volume changes the standard error itself, so the bound should be recomputed from that day's actual sample size, not a fixed number.
• Paging on every statistically significant drop, however small in absolute terms, causes alert fatigue; the critical threshold's larger relative-drop requirement exists specifically to reserve paging for changes large enough to matter operationally, not just statistically.
• Suppressing alerts during known experiments is necessary but risky if the suppression list itself goes stale; audit it periodically so a real incident during a supposed known-experiment window doesn't get silently ignored.

Direct answer

Pick behavioral signals from different parts of the product, usage depth, frequency, outcome achievement, support friction, and breadth of use, so a single tactic can't hide the real risk, and validate each one against actual historical churn on a labeled cohort before wiring it into any early-warning score.

Structured elaboration

The general test protocol: since each of these starts as a candidate, not a finished model, validate it individually against a historical, labeled churn cohort (did users who showed this signal actually churn more often than those who didn't) before combining anything into a composite score.

Worked example

Historical cohort of 5,000 subscription customers observed over a quarter: 400 had a 50% or greater drop in 7-day usage versus their own 30-day baseline at some point; 4,600 did not. Of the 400 flagged, 140 churned within the next 60 days. Of the 4,600 not flagged, 322 churned.

A customer flagged by this usage-drop indicator is five times more likely to churn in the next 60 days than one who isn't flagged, an effect size large enough to be worth operationalizing.

Trade-offs and pitfalls

A relative-risk figure alone ignores base rates: with only 400 of 5,000 customers ever flagged by this indicator, even catching all of them still misses the 322 churners who were never flagged, so precision and recall both need to be reported, not relative risk on its own. Indicators that correlate with each other, login frequency and feature breadth often move together, can double-count in a composite score if combined naively. Thresholds calibrated once will drift as the product and customer base change, so each indicator needs periodic recalibration, not a one-time setup. Finally, an early-warning score used to trigger outreach can change the outcome it's trying to predict (a contacted customer may churn less for reasons unrelated to the original signal), which is why the program should keep a small holdout group that doesn't receive outreach, to separate the signal's predictive value from the intervention's effect.

Direct answer

Build a multiplicative decomposition tree (revenue equals the product of its drivers), then use a log-space decomposition to split the percentage decline exactly across components, and a dollar-level counterfactual (or Shapley-value averaging, for an exact version) to translate each component's share into dollars. The log method is exact for percentages; the simple dollar counterfactual is only a first-order approximation and leaves a residual when components move together.

Structured elaboration

The tree

graph TD
  A[Revenue] --> B[Traffic]
  A --> C[Conversion rate]
  A --> D[Average order value]
  A --> E[Repeat-purchase frequency]

Revenue = Traffic x Conversion Rate x Average Order Value x Repeat-purchase frequency (a retention-driven term: how many times a converting customer buys again within the quarter). Any additional multiplicative driver is added the same way, as one more term in the sum below.

Two attribution methods

1. Log-decomposition (exact, additive): since revenue is a product of factors, the log of revenue is a sum of the logs of the factors. The percentage change in each factor's log, divided by the total log change, gives an exact percentage attribution that always sums to 100%.
2. Dollar counterfactual (approximate): hold every other factor at its baseline value and swap in the new value for one factor at a time. This gives a dollar figure per factor, but when two or more factors move simultaneously, the sum of these first-order dollar contributions will not exactly equal the total dollar delta; the gap is an interaction term. Shapley-value averaging (averaging the marginal contribution of each factor across all possible orderings) removes this residual if an exact dollar split is required.

Worked example

Baseline quarter (Q0): Traffic $T_0 = 500{,}000$ sessions, Conversion $C_0 = 4.0\%$, Average order value $A_0 = \$120$.

Current quarter (Q1): $T_1 = 480{,}000$ (down 4.0%), $C_1 = 3.8\%$ (down 5.0% relative to $C_0$), $A_1 = \$118$ (down 1.67%).

Log decomposition (natural log of each factor's ratio):

Dividing each term by the total gives the percentage-of-decline attribution: Traffic 37.5%, Conversion 47.1%, Average order value 15.4% (sums to 100%).

Dollar counterfactual (holding other factors at Q0 baseline):

The $8,320 gap (about 3.4% of the total delta) is the interaction term: three factors declined simultaneously, and the first-order method double-counts part of that overlap. Reporting the log-based percentage split alongside this residual, rather than silently forcing the dollar figures to add up, is the honest way to present this.

Trade-offs & pitfalls

• The multiplicative decomposition assumes independence between factors; when components move together (a price hike raising average order value while suppressing conversion, for example), a naive first-order dollar attribution misallocates the interaction, exactly as shown by the residual above.
• The log-based percentage split is exact and easy for stakeholders to reason about ("who moved the needle most"), but it does not itself hand you a dollar figure per driver; pair it with the counterfactual or Shapley method when a dollar amount is required.
• Correlated changes need segment-level drill-down (channel, geography, cohort) to find a shared underlying cause rather than treating each branch of the tree as independent.
• A decomposition tree explains where a change occurred, not why; pair it with root-cause investigation, data-quality checks, the pricing and promotion calendar, and competitive or seasonal context, before presenting conclusions to stakeholders.

Direct answer

Pick five guardrails that together cover quality, safety, and operational health, not five variations on the same signal, and decide the alert posture for each by asking two questions: how reversible is the harm, and how much traffic will be affected before a human can react. Metrics where harm compounds quickly (safety, hard outages) get aggressive automated alerting; metrics that are noisy or slow-moving in the first 30 days get observed passively until there is enough data to set a reliable threshold.

Structured elaboration

Two decision criteria drive the aggressive-versus-passive split:

• Reversibility and blast radius. A safety or latency problem harms every user it touches immediately and is hard to walk back after the fact, so it earns an automated, low-latency alert. A coverage dip is recoverable (backfill, re-run the pipeline) and rarely harms a user directly, so it can sit on a dashboard until it crosses a larger multiple of baseline.
• Statistical power in the first 30 days. Early in a rollout, sample sizes are small and day-to-day noise is large. Metrics 1 and 2 need a threshold wide enough to survive that noise (see the worked example), or they will fire constantly and get ignored. Safety metrics don't get this luxury: even one severe incident on a small sample is worth a human look.

Shorter-form variant: if you only have room to name a primary KPI (key performance indicator) plus two guardrails, choose one quality guardrail (metric 1, since it detects the fastest-moving regression) and one safety or operational guardrail (metric 4 or 5), and note that the fuller five-metric set is what you'd expand to once you have room.

Worked example

Suppose the recommender's click-through rate has run at a stable 8.0% mean with a day-to-day standard deviation of 0.6 percentage points across the ten launches this team has instrumented before. To avoid reacting to ordinary daily noise, set the aggressive-alert floor three standard deviations below baseline:

If day 4 of the rollout shows a click-through rate of 6.0%, that is below the 6.2% floor, so the alert fires.

For the safety guardrail, suppose the service serves 100,000 recommendations a day and the historical baseline violation rate is 0.01%, giving an expected count of:

If the safety classifier flags 40 recommendations on day 4, that is:

a four-times jump on a safety metric, which triggers an aggressive alert and immediate human review even though the absolute volume (40 out of 100,000) still looks small.

Trade-offs and pitfalls

Setting every guardrail to aggressive alerting causes fatigue: on-call staff start ignoring alerts, which defeats the purpose of the safety-critical ones. Setting thresholds too tight during the low-traffic early days of a rollout produces false alarms before there is enough data to estimate a stable baseline; widen the statistical window (or delay aggressive alerting) until sample size supports it, rather than shipping a threshold that will cry wolf. Guardrails can also conflict: a team might suppress low-confidence recommendations to protect the safety metric, which quietly drags down coverage and engagement, so the guardrail set needs a person empowered to resolve trade-offs when two of the five move in opposite directions, rather than leaving each metric owner to react in isolation.

Direct answer

Map each AARRR (Acquisition, Activation, Retention, Referral, Revenue, commonly called "pirate metrics") stage to one or two channel-attributable KPIs (key performance indicators) and one visualization built for a fast, non-technical read: a stacked bar for acquisition volume and cost by channel, a funnel for activation drop-off, a cohort heatmap for retention, and a combined trend-plus-bar for revenue. The brief here focuses on acquisition, activation, retention, and revenue; referral would slot in the same way (referral rate, one viral-loop chart) if the marketing manager later asks for it.

Structured elaboration

Include a channel filter and a date-range picker so the same dashboard supports both a monthly executive glance and a campaign manager's weekly diagnostic dig, and annotate major campaign launches directly on the charts so a metric move has an obvious candidate explanation attached.

Worked example

For one channel with $10,000 spent in a month producing 500 new signups:

Of those 500 signups, 350 complete the defined activation action (say, creating a first project):

Of the original 500 signups, 140 are still active on day 7:

A marketing manager reading the funnel sees the full chain at a glance: 500 acquired at $20 each, 70% activate, and 28% of the original cohort are still around a week later, which is a very different story than any single one of those numbers shown alone.

Trade-offs & pitfalls

A single shared dashboard that tries to serve every stage equally for every audience tends to serve none of them well; an executive wants the revenue trend and retention heatmap, while a campaign manager wants the acquisition breakdown and activation funnel drilled down by specific channel and creative. The fix is one dashboard with role-based default views, not one static layout for everyone. A common mistake is choosing acquisition volume as the headline number without pairing it with cost or downstream activation, which rewards channels that bring cheap, low-intent traffic that never activates. Cohort heatmaps communicate retention trends well but can be misread by a non-technical viewer as a simple gradient rather than a time-since-acquisition comparison; a one-line caption stating what the axes mean avoids that misreading.