Metric Definition and Implementation Questions

Direct answer: A 28-day rolling retention metric asks, for each day D, "of users active on day D, what fraction were ALSO active at some point in the trailing 28 days ending on D" (a moving, activity-based window), while cohort retention asks "of users who FIRST joined on a specific date, what fraction are active exactly N days later" (a fixed cohort, anchored to acquisition date).

Structured elaboration and formula:

• 28-day rolling retention: rolling_retention(D) = |{users active on D} ∩ {users active at least once in [D-28, D-1]}| / |{users active on D}|. This measures "of people active today, how many were also engaged recently," independent of when they first signed up.
• Cohort N-day retention: cohort_retention(cohort_date, N) = |{users first-active on cohort_date, also active on cohort_date + N}| / |{users first-active on cohort_date}|. This measures "of people who joined on a specific day, how many are still around N days later," which is the metric used throughout the cohort/retention sub-area earlier in this topic.
• The practical difference: rolling retention is a single time series (one number per day, useful for a health-of-the-business trend line), while cohort retention is a matrix (one curve per cohort, useful for comparing whether NEWER cohorts retain better or worse than older ones, which a single rolling number cannot show).

Efficient computation at scale: maintain a daily active_users(date, user_id) table (as in the DAU/MAU efficiency discussion), then compute rolling retention as a self-join within a bounded 28-day window rather than scanning full history; for cohort retention, maintain the cohort assignment (first-active date per user) once, and incrementally extend each cohort's day-N observation as new daily activity tables land, rather than recomputing the whole cohort matrix from scratch on every run.

Worked example: a product with a stable user base but declining NEW-user quality would show STABLE or improving rolling retention (existing engaged users keep coming back) while cohort retention for RECENT cohorts specifically declines (new users increasingly churn quickly); a single rolling-retention number would miss this entirely, since it doesn't separate "long-time users staying engaged" from "new users failing to stick," which is exactly why both metrics are typically tracked together, not as substitutes for each other.

Trade-offs & pitfalls: Rolling retention can look healthy even while acquisition quality is deteriorating, because a large base of long-tenured loyal users can mask a real problem in the newest cohorts; don't use rolling retention alone as a proxy for "are we acquiring good users," which is specifically what cohort retention is designed to answer.

Direct answer: Bring both teams together to agree on ONE canonical "activations" definition (documenting why the other was rejected, not just declaring a winner), implement it once in the shared metrics layer, migrate both pipelines to read from that single definition, and deprecate the legacy variants on a documented timeline rather than an abrupt cutover.

Structured elaboration, the process:

1. Surface the disagreement concretely: show both teams the ACTUAL numeric gap between "first 7-day key event" and "first login + profile completion" on the same historical data, and what decisions each number has been driving, so the conversation is about impact, not abstract preference.
2. Reconcile to one canonical definition: this usually isn't picking one side wholesale; it may be a genuinely new definition informed by both (e.g., "first 7-day key event, where profile completion IS one of the qualifying key events" if that's what both teams' underlying intents actually converge on), decided by whoever has ultimate accountability for the business outcome activations is meant to represent.
3. Implement once: register the canonical definition in the metrics layer (S58/S59's registry/dbt pattern) with a single metric_id, so both product's and growth's pipelines read from the SAME underlying model rather than maintaining two independently-evolving copies.
4. Migrate consumers: update dashboards, alerts, and any downstream models (e.g., an activation-driven onboarding experiment analysis) to point at the new canonical metric, running the OLD and NEW definitions side by side for a transition window so any consumer relying on a specific legacy number can see exactly how and when it will change.
5. Deprecate legacy variants: mark the old definitions deprecated with a superseded_by pointer (S38's registry schema) rather than deleting them outright, so historical reports referencing the old name remain interpretable, and set a firm sunset date communicated well in advance.

Worked example: if "product's" and "growth's" definitions of activation disagree by 15% on a sample month, and investigation shows the gap is almost entirely users who complete profile setup without an early key event (counted by growth, not by product), the canonical definition decision is really a business judgment call about whether profile completion alone should count as "activated," not a technical dispute at all; framing it as a business decision (with the data laid out clearly) rather than a data-quality argument is what actually resolves it, since neither original definition was technically WRONG.

Trade-offs & pitfalls: Migrating all downstream consumers to a new canonical definition simultaneously is risky (any consumer with an undiscovered hard dependency on the OLD number's specific behavior breaks all at once); running both in parallel for a documented transition window, with active outreach to known consumers, minimizes the disruption this question specifically asks to minimize, at the cost of a period where two numbers legitimately coexist and need to be clearly labeled to avoid re-introducing the exact confusion this whole process exists to resolve.

Direct answer: A metric is a quantitative measurement derived from data (a number that changes over time and can be tracked, like conversion_rate), a dimension is a categorical attribute you can slice a metric by (like user_country), and a KPI is a metric that has been elevated to a target with a business threshold attached (a metric someone is accountable for hitting a number on).

Structured elaboration:

• Metric vs dimension, concretely: page_view_count is a metric (a count you aggregate); user_country is a dimension (a label you group or filter the metric by, e.g., "page views BY country"). The same underlying event row contributes to both: the row itself is one page view (metric contribution), and it also happens to have been generated by a user in a specific country (dimension value).
• Metric vs KPI: conversion_rate is a metric whether or not anyone has set a target for it; it becomes a KPI when leadership says "we are targeting 4.5% conversion rate this quarter and someone owns hitting it." The same number, different organizational weight.
• Event-level vs aggregated: compute at event-level when the question is about individual occurrences (e.g., "which specific sessions converted"); aggregate to daily/user-level when the question is about a rate or trend over a population (e.g., "conversion rate this week").

Worked example: an e-commerce site logs a page_view event with user_id, country, page_type. page_view_count grouped by nothing is a metric (total volume); grouped by country uses country as a dimension to slice that same metric ("page views per country"); if leadership sets a target of "20% month-over-month growth in page views from a new market," page_view_count for that market becomes a KPI.

Trade-offs & pitfalls: Computing at the wrong level causes real bias: a conversion-rate metric computed at event-level (dividing total purchase events by total view events, both event counts) will be wrong if a single user can view the same page multiple times before converting once; that inflates the denominator relative to a user-level definition (unique purchasers / unique viewers), understating the true rate. The fix is not "always aggregate to user-level"; it's to be explicit about which grain the business question is actually asking, and to name that grain in the metric's documentation so nobody re-derives it inconsistently.

Direct answer: Tumbling windows are fixed, non-overlapping calendar periods (a calendar day, a calendar month); sliding windows roll continuously (a trailing 7-day average recomputed every day); session windows are defined by activity gaps rather than a fixed clock. Choose a rolling 7-day average over month-to-date when you want a stable, noise-smoothed trend that updates daily; choose month-to-date when the business genuinely cares about a fixed accounting period (billing, reporting cadence).

Structured elaboration:

• Tumbling: simplest to reason about and to partition (each period is independent, no overlap to recompute), but can produce a big visible jump at the boundary (day 1 of a new month resets to a small base) that isn't a real behavior change, just an artifact of the fixed window resetting.
• Sliding (rolling): smooths day-to-day noise and avoids the reset artifact, at the cost of being harder to partition efficiently (each day's output depends on a moving set of prior days) and being less intuitive to a stakeholder who thinks in calendar months for planning purposes.
• Session-based: appropriate when the metric is fundamentally about a unit of engaged activity rather than a fixed clock period (as in S23's conversion-rate-per-session); not a substitute for tumbling/sliding when the business question is genuinely calendar-anchored (e.g., "this month's revenue").
• Recommendation: a rolling 7-day average suits a noisy daily operational metric (e.g., daily active users, which has strong day-of-week seasonality) where the goal is trend-watching; month-to-date suits a metric tied to a business/billing cycle where the goal is a specific period's total, and where partial-period comparisons ("we're at 60% of last month's total on day 18") are themselves meaningful.
• Timezone/DST pitfalls: a tumbling daily window computed with fixed 24-hour arithmetic silently mishandles the DST transition day (23 or 25 real hours), either dropping or double-counting an hour of activity; use calendar-aware date truncation, not fixed-duration bucketing, to avoid this. A rolling 7-day window crossing a DST boundary has the same latent issue if implemented as now() - 7*24 hours instead of date subtraction of 7 calendar days.

Worked example: a DAU metric with strong weekday/weekend seasonality looks noisy and hard to eyeball as a raw daily tumbling number; a rolling 7-day average smooths that seasonality out and makes a genuine trend change visible days sooner than waiting for a full month-to-date comparison would.

Trade-offs & pitfalls: Don't silently switch between these mid-series; each produces a genuinely different-shaped line even for identical underlying data, and a switch reads as a step change in trend if not explicitly flagged in the metric's documentation and on the chart itself.

Direct answer: Show both together by default: a percentage without its absolute base can make a tiny change look dramatic (or a huge change look trivial), and denominator instability (a base count that itself fluctuates) makes a percentage-only view actively misleading over time.

Structured elaboration:

• When a percentage misleads: a metric going from 1 to 3 occurrences is "+200%," which reads as an alarming trend, but is plausibly random noise on a tiny base; conversely a metric moving from 40% to 41% might look unremarkable but represent a very large absolute change if the base population is huge (millions of users).
• Denominator instability: if the population you're computing a rate against is itself changing size significantly (e.g., a rapidly growing user base), a stable percentage can mask a rapidly growing (or shrinking) absolute number, since the percentage only tells you the RATIO, not the scale it's a ratio of.
• Recommendation: present the percentage alongside the absolute numerator and denominator explicitly ("42% (420 of 1,000)" rather than bare "42%"), and when comparing periods, show the percentage-point CHANGE and the absolute change together, since a viewer needs both to correctly judge whether a change is meaningful.

Worked example: a conversion metric for a small new market segment might show "conversion rate up 200% week over week," which sounds dramatic; showing the underlying numbers, "up from 2 to 6 conversions out of a base of roughly 40 visitors," immediately recontextualizes it as plausible small-sample noise rather than a real trend, without needing any statistical test to make the point to a general audience.

Trade-offs & pitfalls: Always presenting both is more useful than a rule trying to decide case by case when to show which; the discipline of habitually pairing percentage with absolute numbers costs almost nothing in dashboard real estate and prevents most of the misleading-percentage complaints this question is really about. The remaining judgment call is which one to lead with visually: for an audience making resourcing decisions (should we invest more here), lead with the absolute number, since that's what resourcing scales with; for an audience comparing efficiency across differently-sized segments, lead with the percentage, since that's the only fair basis for comparison across segments of different size.