Design Metrics and Impact Measurement Questions

Direct answer

To claim a redesign CAUSED a change in revenue or retention, not just that the two happened around the same time, the study needs either true randomization (an A/B test, the strongest design) or, when a full redesign genuinely cannot be randomized at the user level, a quasi-experimental design like difference-in-differences with a matched control population. Either way the plan needs a pre-registered hypothesis, a power calculation that sets the required sample size before launch, explicit control variables for the confounders that would otherwise explain the same result, and a statistical test whose assumptions actually hold for the design chosen.

Structured elaboration

Hypotheses: state them before looking at any data. Null (H0): the redesign has no effect on 30-day retention or 30-day revenue per user. Alternative (H1): the redesign increases 30-day retention by at least a pre-specified amount and revenue per user by at least a pre-specified amount. Naming the minimum effect you care about up front (not "any effect") is what makes the later power calculation possible.

Experimental design, in order of preference:

1. Randomized controlled experiment (RCT): randomize at the user level into redesign versus legacy UI behind a feature flag, and keep a small permanent holdout (for example 5 to 10%) even after full rollout, so there is always a live counterfactual to compare against later.
2. Stepped-wedge rollout: if full randomization is operationally impossible (for example the redesign touches shared backend state that can't cleanly serve two UI versions at once), randomize WHICH cohort (a group of users defined by something they share, like the week they signed up) or market gets the redesign and WHEN, so everyone eventually gets it but you still get both treated and not-yet-treated data to compare at each point in time.
3. Difference-in-differences (DiD) with a matched control: if you can't randomize at all (say, the redesign ships to everyone on iOS at once), find a comparable, un-treated population (for example, Android, if its release lags) and compare the CHANGE in each metric before and after launch, on the assumption that both groups would have moved in parallel absent the redesign. This assumption, called the parallel-trends assumption, has to be checked against pre-period data before trusting the result, not assumed.

Metrics: primary metrics are 30-day revenue per user (or ARPU, average revenue per user) and 30-day retention; guardrail metrics (metrics you are not trying to move, but must not let get worse) are churn rate, support ticket volume, page load time, and NPS (Net Promoter Score, a self-reported loyalty measure), so a revenue win that comes with a support-ticket spike or an NPS drop doesn't get called an unqualified success.

Sampling and power: the sample size has to be big enough to detect the smallest effect you'd actually act on, not just any effect. For a baseline 30-day retention of 25% and a target minimum detectable lift of 2 percentage points (to 27%), at a significance threshold (alpha) of 0.05 two-sided and 80% statistical power (the chance of detecting the effect if it's really there): here z_alpha/2 is the z-score cutoff for that significance threshold (1.96 for a two-sided alpha of 0.05) and z_beta is the z-score for that power target (0.84 for 80% power):

That's roughly 15,100 users total, before accounting for any additional segmentation.

Control variables (confounders to account for): acquisition channel mix (a paid-campaign spike during the test window changes user quality independent of the redesign), device/platform mix, user tenure, geography (regulatory or pricing differences), and calendar effects (holidays, paydays). In the RCT, randomization handles these automatically as long as the sample is large enough; in the DiD design, they have to be added explicitly as regression covariates, since the two groups being compared were never randomly assigned in the first place.

Statistical tests: for the RCT, a two-proportion z-test (or two-sample t-test for the continuous revenue metric) comparing treatment and control directly. For the DiD design, a linear regression with an interaction term between "treatment group" and "post-period," where the coefficient on that interaction IS the causal estimate, with standard errors clustered at the market or cohort level (not the individual user level) since users within the same market share correlated shocks.

Worked example

Continuing the retention example with illustrative results: with roughly 7,538 users per arm and an observed treatment retention of 27.4% against a control of 25.1% (a 2.3 point lift), the pooled proportion is:

SE (standard error) here is the amount of random wobble expected between two samples of this size purely from chance, if the true retention rate were identical in both groups; z restates the observed 2.3-point gap as a multiple of that expected wobble, so a z of 3.21 means the gap is more than three times larger than the noise you would expect under pure chance. That exceeds the z_alpha/2 = 1.96 cutoff for the two-sided alpha = 0.05 threshold set above, so this illustrative result clears the pre-specified significance bar, and because it came from a randomized design (not the DiD alternative), it also supports a causal claim, not just a correlational one.

Trade-offs and pitfalls

Quasi-experimental designs like DiD have weaker internal validity than a true RCT: if the parallel-trends assumption is wrong (the control market was already diverging before launch for unrelated reasons), the estimate is biased and there's no statistical test that fully rescues that, only a placebo check on pre-period data that makes the assumption more or less credible. Randomizing at the user level for a big visual redesign also risks contamination if users interact with each other (a social feature where one user's experience leaks into another's feed), which biases a naive comparison toward showing no effect even when a real one exists; that risk is usually handled by randomizing at a cluster level (market, or friend-group) instead of the individual. Finally, picking a control market because it happened to be "similar" after the fact is a common trap, since a market chosen for looking stable can regress to the mean regardless of the redesign, which is exactly why the pre-period parallel-trends check has to happen before, not after, seeing the result.

Direct answer. The schema has no separate "signup" event, so the cohort key has to be derived: define cohort_week as the Monday-starting calendar week of each user's first session event, then for every later week count how many of that cohort's users had at least one more session event, expressed as week_offset (0 through 11 weeks after the cohort week).

Approach.

1. Find each user's first session date and truncate it to the week: that is cohort_week.
2. Build a de-duplicated set of (user, week) pairs where the user had a session in that week.
3. Join each user's cohort_week to every week they were active on or after it, and compute week_offset as the number of 7-day blocks between the two weeks.
4. Cross-join every cohort against week_offset 0..11 with generate_series, so weeks with zero returners still appear as a row instead of being silently missing.
5. Count distinct returning users per (cohort_week, week_offset) and divide by cohort size.

WITH first_activity AS (
  SELECT user_id, date_trunc('week', MIN(event_date))::date AS cohort_week
  FROM events
  WHERE event_name = 'session'
  GROUP BY user_id
),
activity_weeks AS (
  SELECT DISTINCT user_id, date_trunc('week', event_date)::date AS activity_week
  FROM events
  WHERE event_name = 'session'
),
cohort_activity AS (
  SELECT f.user_id, f.cohort_week,
         ((a.activity_week - f.cohort_week) / 7)::int AS week_offset
  FROM first_activity f
  JOIN activity_weeks a ON a.user_id = f.user_id
  WHERE a.activity_week >= f.cohort_week
    AND ((a.activity_week - f.cohort_week) / 7)::int <= 11
),
cohort_sizes AS (
  SELECT cohort_week, COUNT(DISTINCT user_id) AS users_in_cohort
  FROM first_activity
  GROUP BY cohort_week
),
offsets AS (
  SELECT generate_series(0, 11) AS week_offset
),
cohort_offsets AS (
  SELECT cs.cohort_week, o.week_offset, cs.users_in_cohort
  FROM cohort_sizes cs
  CROSS JOIN offsets o
)
SELECT
  co.cohort_week,
  co.week_offset,
  co.users_in_cohort,
  COUNT(DISTINCT ca.user_id) AS returning_users,
  ROUND(COUNT(DISTINCT ca.user_id)::numeric / co.users_in_cohort, 4) AS retention_rate
FROM cohort_offsets co
LEFT JOIN cohort_activity ca
  ON ca.cohort_week = co.cohort_week AND ca.week_offset = co.week_offset
GROUP BY co.cohort_week, co.week_offset, co.users_in_cohort
ORDER BY co.cohort_week, co.week_offset;

Worked example. Run against a pinned events table with two 3-user cohorts (week of 2026-01-05 and week of 2026-01-12), including one duplicate same-day event for u1 and one non-session event for u2 before their real first session (to prove the filter ignores it):

CREATE TABLE events (user_id text, event_name text, event_date date);
INSERT INTO events (user_id, event_name, event_date) VALUES
('u2','click','2026-01-01'),
('u1','session','2026-01-05'), ('u1','session','2026-01-05'),
('u1','session','2026-01-12'), ('u1','session','2026-01-20'),
('u2','session','2026-01-06'),
('u5','session','2026-01-07'), ('u5','session','2026-01-14'),
('u3','session','2026-01-13'), ('u3','session','2026-01-19'),
('u4','session','2026-01-15'),
('u6','session','2026-01-16'), ('u6','session','2026-01-21'), ('u6','session','2026-01-28');

Running the query above against exactly this data produces (week_offset 3-11 are all zero for both cohorts, so shown once and noted rather than repeated):

This matches by construction: for the first cohort, u1 and u5 return in week 1 (2/3 = 0.6667) and only u1 returns in week 2 (1/3 = 0.3333); u2's pre-cohort click event never enters the calculation because it isn't a session row.

Complexity. Two filtered scans of events for session rows, a bounded join (cohort size times 12 offsets, not a full self-join across all users), and a hash aggregate. With an index on (event_name, user_id, event_date) this is a couple of index scans plus a sort, not a quadratic operation, and it stays cheap even as the table grows, since the join fan-out is capped at 12 rows per user rather than growing with total row count.

Edge cases.

• Right-censoring on recent cohorts: only an offset whose week has fully elapsed carries information. A (cohort_week, week_offset) pair is reportable only once cohort_week + (week_offset + 1) * interval '7 days' is in the past, which means a cohort that is k weeks old supports offsets 0 through k-1 and nothing beyond. So for a cohort formed a week ago (k = 1), only week_offset 0 has finished: week_offset 1 is the week still in progress and week_offset 2 through 11 lie entirely in the future, so every one of offsets 1 through 11 reads as 0 by construction, even though those users simply haven't had the chance to return yet, which is different from having churned. In production, filter those pairs out with exactly that predicate, or clearly footnote them, rather than presenting them as real zero retention; note that this censoring bites hardest on the newest cohorts, which are also the ones a stakeholder is most eager to read, so the guard has to live in the query rather than in a caveat someone can skip.
• Duplicate events: two session rows for the same user on the same date must not double-count a return; the DISTINCT in activity_weeks and the COUNT(DISTINCT ...) in the final SELECT guard against this, as the duplicate u1 row on 2026-01-05 shows.
• Timezones: event_date is a plain date column with no timezone attached to it in this query. If the underlying event timestamps carry a timezone and get truncated to date upstream, a user active right at a week boundary can land in the wrong week depending on which timezone that truncation used; verify the ETL truncates consistently (usually to the organization's canonical timezone, not each user's local one) before this query ever sees the data.

Direct answer. Do not diagnose this from the topline number: predefine a handful of segments where the redesign plausibly interacts differently with user behavior, confirm the split is real (not noise) with an interaction-term analysis, localize the specific broken step for the harmed segment, propose a segment-scoped mitigation such as an advanced-mode toggle rather than reverting the whole redesign, and validate that mitigation with its own experiment scoped to just that segment.

Analysis plan to identify affected segments.

1. Predefine 4 to 6 plausible segment cuts before looking at the data, based on product hypotheses, not after browsing dozens of possible splits: user tenure/power-user tier (for example, top-quartile by monthly actions), primary workflow used, device or platform, and plan tier. Pre-registering the cuts and applying a multiple-comparison correction (such as Bonferroni, which simply divides your significance threshold by the number of segments tested, so testing 5 pre-registered segments at the usual alpha = 0.05 means each one now needs p < 0.01 to count as significant) protects against declaring a segment "affected" purely from noise created by testing many splits at once.
2. Fit an interaction model, outcome ~ treatment * segment (read the tilde as "modeled as a function of" and the asterisk as "include treatment on its own, segment on its own, and a combined interaction term for the two together"), and look specifically at the treatment-by-segment interaction coefficient, the term that isolates whether the treatment effect itself differs by segment, not just the two segments' raw before/after numbers side by side; confirm the confidence intervals for the two segment effects genuinely don't overlap before concluding the divergence is real rather than sampling noise.
3. For the confirmed power-user segment, drill into funnel-step-level metrics (not just final conversion) to localize exactly where they drop off, and check session recordings or error logs for that step to distinguish a removed capability, an added-friction step, and a performance regression, three different problems with three different fixes.

Mitigations per affected segment. For power users hurt by a simplified flow, the common fix is an advanced or pro mode, a settings toggle that restores the removed capability, rather than reverting the redesign wholesale and losing the gain among new users. New users, who are unaffected or improved, typically need no mitigation at all; resist the urge to "fix" a segment that isn't broken just because the overall story feels uncomfortable.

Measurement to validate the mitigation. Run the proposed fix as its own randomized experiment scoped only to the affected segment, using the same segment-level metric that originally flagged the regression as the primary metric, plus the original topline (or the new-user metric) as a guardrail to confirm the fix does not erode the gain among the segment that was already doing better.

Worked example. Suppose traffic mix is 70% new users and 30% power users, held constant before and after (so any change in the blended number is a pure weighted average of the two segment effects, not a mix-shift artifact). Before the redesign, new users convert at 8% and power users at 40%. After, new users convert at 10% (+2 percentage points) and power users at 34% (-6 percentage points):

$\text{blended}_{\text{pre}} = 0.7 \times 0.08 + 0.3 \times 0.40 = 0.176 = 17.6\%$
$\text{blended}_{\text{post}} = 0.7 \times 0.10 + 0.3 \times 0.34 = 0.172 = 17.2\%$

The blended metric actually decreases slightly (17.6% to 17.2%), even though the description ("new users up, power users down") sounds like a wash. A team only watching the topline could misread this as "the redesign roughly failed," when it in fact clearly helped one identifiable group and hurt another by more in absolute weighted terms. This is why the segment-level split, not the topline, is where diagnosis and action both need to happen: the heterogeneous treatment effect (HTE, an effect whose size or direction genuinely differs across subgroups) here is real and large, and it is being hidden by a single blended average.

Trade-offs and pitfalls. Carving too many post-hoc segments inflates the risk of a false "affected segment" finding; guard with pre-registration and a multiple-comparison correction, not intuition. A segment-specific fix can itself add cost, two different experiences for two user types raises maintenance burden and can confuse a user who transitions from new to power over time. Don't declare victory purely on the sub-experiment's own metric, since a scoped fix can still have spillover effects (for instance, a toggle implemented sloppily can complicate the interface for everyone), so re-check the topline and the unaffected segment's guardrail metric too (a secondary metric you watch specifically to catch an unintended side effect, even when the metric you're optimizing for looks fine) before rolling the mitigation out broadly.

Direct answer

A flat-revenue, higher-conversion read after a traffic shock is ambiguous by construction: it could mean the redesign or another change genuinely made the surviving users convert better, or it could just mean the users who kept showing up are a different, higher-intent mix (a selection effect) than before. You isolate the two by checking whether conversion moved WITHIN comparable segments, not just in the blended average, and by finding a comparable population the shock did not hit to use as a counterfactual.

Structured elaboration

1. Pin the timeline first. Get the exact date/time of the external event and check whether any design change, feature flag (a toggle that switches a feature on or off for some users without shipping new code, often used to roll changes out gradually), or live experiment was active in the same window. This matters more than it sounds: if an A/B test happened to be running when the shock hit, and the shock hit the treatment and control arms unevenly (for example, one arm's traffic source was more exposed to the event), your randomization is already broken before you even start the selection-effect analysis.
2. Segment before you average. Break traffic and conversion down by acquisition channel, device, new versus returning, and geography, before and after the event. If the conversion lift disappears once you hold segment mix fixed, that is the fingerprint of a pure selection effect: the same kinds of users are converting at the same rate they always did, there are just proportionally more high-converting segments left.
3. Regression adjustment. Fit conversion as a function of the observable segment covariates plus a post-event indicator. If the coefficient on the post-event indicator shrinks toward zero once segment mix is controlled for, most of the apparent improvement is compositional, not causal.
4. Propensity score matching (PSM). PSM pairs each post-event session with a pre-event session that had a similar estimated probability of belonging to the post-event group, based on observable traits like channel, device, and day of week. Comparing conversion within these matched pairs removes bias that a raw before/after average would carry from the mix shift.
5. Difference-in-differences (DiD). Find a comparable population the shock did not touch (a different market, property, or channel with a similar historical trend). Compute the change in conversion in your affected group and the change in the unaffected control group over the same period; the gap between those two changes is your estimate of anything beyond a shared trend, i.e. something specific to the affected population.
6. Sanity checks. Look at absolute conversion counts, not just rates (did conversions actually rise or did the denominator collapse), and directionally confirm that low-converting segments actually shrank as a share of traffic, since that is what "selection" should look like on the ground.

Worked example

Illustrative decomposition, not real data, showing the mechanism. Before the event: 100,000 visits/day, split 20% high-intent segment (converts at 8%) and 80% low-intent (converts at 2%). Conversions = 20,000(0.08) + 80,000(0.02) = 1,600 + 1,600 = 3,200, a 3.2% blended rate.

After the event, the low-intent channel is hit hardest: 60,000 visits/day, now split evenly 50/50 between the two segments, with the PER-SEGMENT rates unchanged (still 8% and 2%). Conversions = 30,000(0.08) + 30,000(0.02) = 2,400 + 600 = 3,000, a 5.0% blended rate. That is a +56% relative jump in the headline conversion rate, purely from mix shift, while total conversions actually fell slightly (3,200 to 3,000) and per-segment behavior never moved at all. A regression with segment share as a covariate would correctly show the post-event coefficient at roughly zero here: there is no residual to attribute to the redesign, and a DiD against an untouched control region (whose segment rates also stayed flat) would confirm it.

Trade-offs and pitfalls

PSM only removes bias on covariates you actually observed; unmeasured intent differences can still leak through. DiD depends on the parallel-trends assumption, that your control population would have moved the same way absent the shock, which you cannot prove, only argue for with historical trend data. If a real design change shipped in the exact same week as the external event, you may not be able to cleanly separate the two effects without a genuine holdout; in that case the honest conclusion is "directionally suggestive, not conclusive," paired with a recommendation to protect a proper holdout on the next related change rather than a confident causal claim you cannot defend.

Direct answer

Compute the required sample size per variant with the standard two-proportion z-test formula, plug in your baseline rate, target rate, and the significance level (alpha, the false-positive rate you're willing to accept) and power (the probability of detecting the effect if it's real), then divide that sample size by your expected daily traffic per variant to get the test duration.

Structured elaboration

1. Name the inputs. Baseline conversion $p_1 = 0.04$, target conversion $p_2 = 0.044$ (a 10% relative lift), alpha $= 0.05$ two-sided (so $z_{\alpha/2} = 1.96$), power $= 0.80$ (so $z_{\beta} = 0.84$, the z-score for detecting a true effect 80% of the time).
2. Compute the average rate. $\bar p = (p_1+p_2)/2 = 0.042$, used for the variance under the null hypothesis of no difference.
3. Apply the formula:

The intuition: the numerator measures how much combined noise (from both the null-hypothesis variance and the true-effect variance) you need to overcome, and the denominator is the squared size of the effect you're trying to detect, so the smaller the effect, the more samples you need, and it grows with the square of that shrinkage.

4. Substitute and solve. $2\bar p(1-\bar p) = 0.0805$, so that term is $1.96\sqrt{0.0805} \approx 0.556$. $p_1(1-p_1)+p_2(1-p_2) = 0.0384 + 0.0421 = 0.0805$, so that term is $0.84\sqrt{0.0805} \approx 0.239$. Sum $\approx 0.795$, squared $\approx 0.632$. Divide by $(p_2-p_1)^2 = 0.004^2 = 0.000016$: $n \approx 39{,}475$ per variant (round up to be conservative).
5. Translate to duration. With 20,000 daily unique visitors split 50/50, that's 10,000 per variant per day. Duration $= 39{,}475 / 10{,}000 \approx 3.95$ days, so about 4 days to reach the statistical minimum.

Worked example

The numbers above are the worked example: baseline 4.0%, target 4.4%, alpha 0.05, power 0.80 gives roughly 39,500 users per variant (79,000 total), reachable in about 4 days at 20,000 daily visitors split evenly.

Trade-offs and pitfalls

Never actually stop a test at the bare statistical minimum duration. Four days does not cover a full weekly cycle, so weekday/weekend behavior differences would bias the result; the practical rule most teams use is to round the statistical minimum UP to the nearest full week (or two, to average out any single unusual week) rather than shipping the decision the moment the sample-size math is satisfied. Also watch for a mismatch between the assumed 50/50 split and the actual traffic allocation, and remember this formula assumes independent, non-repeating users; if the same visitor can convert more than once, or sessions cluster by user, the effective sample size is smaller than the raw visit count suggests.