Design Metrics and Impact Measurement Questions

Direct answer

P-hacking and multiple-comparisons false positives both come from the same root cause: testing many things and reporting only the ones that happened to look significant. The fix is not a single statistical trick, it's a program with four legs that each close off a different way this happens: a policy that locks the hypothesis and primary metric before launch, tooling that enforces that policy and corrects for the number of comparisons actually made, a cultural norm that treats a single "surprising" result as a hypothesis to re-test rather than a finding to ship, and a habit of publishing null results, not just wins.

Structured elaboration

Policy:

• Require every experiment to pre-register one primary metric and hypothesis before launch; anything else measured is labeled secondary or exploratory and cannot alone justify a launch decision.
• Fix the sample size and stopping rule (a target duration or a target number of users) before launch, and prohibit stopping early just because the metric looks favorable that day (repeated "peeking" without a correction inflates the false-positive rate well above the nominal 5%).
• Ban HARKing (Hypothesizing After the Results are Known: writing the hypothesis to match a pattern you already found in the data) by requiring the hypothesis document to be timestamped before the experiment starts.

Tooling:

• Build sample-size and minimum-detectable-effect calculators directly into the experimentation platform so nobody launches without knowing how long the test needs to run for its stated effect size.
• Apply a multiple-comparisons correction automatically whenever more than one metric or segment is being evaluated for a launch decision, rather than leaving each analyst to remember to do it by hand.
• Tag every analysis as confirmatory (pre-registered, feeds a decision) or exploratory (hypothesis-generating, needs a follow-up confirmatory test) in the platform itself, and visually distinguish them on dashboards so a raw exploratory p-value can't be mistaken for a decision-grade one.

Statistical adjustments, worked example:

Testing 20 independent metrics at a significance level (alpha, the false-positive rate you're willing to accept per test) of 0.05 each, the chance of at least one false positive by pure chance is:

So a team testing 20 metrics per experiment should expect roughly a 64% chance that at least one of them looks "significant" by chance alone, even if the change did nothing. The Benjamini-Hochberg procedure controls the expected proportion of false positives among the ones you call significant (the false discovery rate) instead of ignoring the problem. For 5 metrics sorted by p-value at alpha = 0.05:

Raw alpha = 0.05 would have called the first four metrics "significant" since each p-value is below 0.05 on its own. Benjamini-Hochberg only confirms rank 1, and the direction the procedure runs in matters, because this
is the part people routinely get backwards. Benjamini-Hochberg is a step-UP procedure: scan from the
LARGEST p-value downward, find the largest k whose p-value is at or below its k/m x alpha threshold,
and declare significant every metric ranked at or below that k, including any that failed its own
individual threshold along the way. Here rank 5 (0.200 > 0.050), rank 4 (0.045 > 0.040), rank 3
(0.040 > 0.030) and rank 2 (0.030 > 0.020) all fail, so the largest passing k is 1 and only rank 1 is
confirmed. The opposite case shows why the direction matters: if rank 4 had come in at p = 0.038
instead of 0.045, then k = 4 would pass (0.038 <= 0.040) and ranks 1 through 4 would ALL be declared
significant, including rank 2 at 0.030 and rank 3 at 0.040, both of which fail their own thresholds. A
failure at a smaller k never blocks a larger k from passing, and a failure at a larger k never
retroactively cancels a smaller one; only the largest passing k does any work. This is also why
Benjamini-Hochberg is less brutal than Bonferroni, which would have compared every p-value against a
flat 0.05/5 = 0.010 and confirmed rank 1 and nothing else no matter how the other four landed. On the
numbers as given, the concrete cost of not correcting is that four apparent wins shrink to one.

Cultural and process changes:

• Require a second, independent replication (a holdout re-run) before any launch decision that rests on a secondary or exploratory metric, never ship on a single exploratory hit.
• Keep a shared, searchable log of every experiment run, including nulls and losses, so the organization can see the true base rate of "wins" rather than only hearing about the successes (avoiding the file-drawer effect, where negative results quietly disappear).
• Route any experiment whose design or analysis plan changed mid-flight through a lightweight peer review before the result is used to justify a decision.

Trade-offs and pitfalls

Correcting for multiple comparisons trades false positives for false negatives: a real effect can get buried under a strict correction, especially with many metrics, so teams sometimes reserve a small number of pre-declared primary metrics that get standard alpha = 0.05 treatment and apply the correction only to the larger pool of secondary metrics. The cultural piece is the one teams skip because it's the hardest to enforce technically: no tool can stop someone from quietly re-running "just the segment that looked good," which is why a norm of treating segment cuts as hypothesis-generating, and requiring a fresh confirmatory test before acting on them, matters as much as the statistics.

Direct answer

Testing 5 headline variants against a control means running 5 separate hypothesis tests in one experiment, and each one carries its own 5% chance of a false positive at the usual significance threshold (alpha = 0.05, the accepted false-positive rate per test). Run enough tests and the chance that at least one comes back "significant" purely by luck climbs well past 5%. The frequentist fix is a Bonferroni correction; the modern, less conservative alternative is Benjamini-Hochberg false discovery rate (FDR) control.

Structured elaboration

• The multiple comparisons problem: with m independent tests each at significance level alpha, the family-wise error rate (probability of at least one false positive across the whole family) is:

• Bonferroni correction (frequentist): divide alpha by the number of tests, so each individual test must clear a stricter bar:

It is simple and guarantees the family-wise error rate stays at or below alpha, but it is conservative: it treats every comparison as if it must survive the worst case, which costs statistical power (the chance of detecting a real effect when one exists) as the number of variants grows.

• Benjamini-Hochberg (modern alternative): instead of controlling the chance of any false positive, it controls the expected proportion of false positives among the discoveries you actually declare (the false discovery rate). Sort the m p-values from smallest to largest, and find the largest rank i where:

Here Q is the false discovery rate you are willing to tolerate, the target share of your 'significant' results that are allowed to be false positives. It plays a similar role in this formula to the one alpha (the per-test false-positive rate) played above, and teams commonly set Q to the same 0.05 value used for alpha elsewhere, though the two control different things: alpha bounds the chance of any false positive at all, while Q bounds the expected proportion of false positives among the results you actually declare as discoveries.

Reject every hypothesis at rank 1 through that largest i. This is less conservative than Bonferroni, so it finds more real effects at the cost of tolerating a small, known rate of false discoveries among the winners.

Worked example

Take 5 variant-vs-control tests coming back with p-values 0.001, 0.009, 0.019, 0.031, and 0.062 (illustrative numbers, chosen to show the mechanics). With m = 5 at alpha = 0.05, the family-wise error rate before any correction would be 1 - 0.95^5 = 22.6%, over four times the nominal 5%. Bonferroni sets the per-test bar at 0.05 / 5 = 0.01, so only the two smallest p-values (0.001 and 0.009) clear it. Benjamini-Hochberg checks each p-value against (i/5) x Q, using Q = 0.05 here, the same numeric value as alpha in the Bonferroni comparison above (chosen so the two methods are directly comparable in this example, though Q and alpha remain conceptually different targets): rank 4's p-value of 0.031 clears its threshold of (4/5) x 0.05 = 0.04, so it rejects ranks 1 through 4 (everything except the p = 0.062 variant), finding two more real winners than Bonferroni would have.

Trade-offs and pitfalls

Bonferroni is the right call when a false positive is expensive to walk back (a pricing or legal-facing change), because it minimizes the chance of shipping any wrong winner. Benjamini-Hochberg is the right call for a lower-stakes decision like a headline, where finding more genuine winners matters more than an occasional false one, since you can always re-test a surprising result before fully committing. The trap either way is running the correction after peeking at the data repeatedly during the test: both methods assume the p-values were generated once, not checked daily until one looked good.

Approach

Build one CTE (Common Table Expression, a named temporary result defined with a WITH ... AS clause) per funnel step that requires the step's event to occur strictly after the previous step's qualifying event for the same user, using MIN(timestamp) at each stage so a repeated event only counts once and an out-of-order event is correctly excluded. Filter to the last 30 days on the event side, then left join the per-user step flags to the full users table and group by each user's acquisition week (from users.created_at), the cohort each user belongs to, to get step counts, step-to-step conversion, and overall conversion.

SQL

WITH params AS (
  SELECT timestamptz '2026-09-01 00:00:00+00' AS as_of  -- swap for now() in production
),
recent AS (
  SELECT e.* FROM events e, params p
  WHERE e.timestamp >= p.as_of - interval '30 days' AND e.timestamp < p.as_of
),
visits AS (
  SELECT user_id, MIN(timestamp) AS visit_ts
  FROM recent WHERE event_name = 'visit'
  GROUP BY user_id
),
searches AS (
  SELECT v.user_id, MIN(e.timestamp) AS search_ts
  FROM visits v
  JOIN recent e ON e.user_id = v.user_id AND e.event_name = 'search' AND e.timestamp > v.visit_ts
  GROUP BY v.user_id
),
views AS (
  SELECT s.user_id, MIN(e.timestamp) AS view_ts
  FROM searches s
  JOIN recent e ON e.user_id = s.user_id AND e.event_name = 'view_product' AND e.timestamp > s.search_ts
  GROUP BY s.user_id
),
purchases AS (
  SELECT vw.user_id, MIN(e.timestamp) AS purchase_ts
  FROM views vw
  JOIN recent e ON e.user_id = vw.user_id AND e.event_name = 'purchase' AND e.timestamp > vw.view_ts
  GROUP BY vw.user_id
),
funnel AS (
  SELECT
    u.user_id,
    date_trunc('week', u.created_at AT TIME ZONE 'UTC') AS acquisition_week,
    (v.user_id IS NOT NULL) AS did_visit,
    (s.user_id IS NOT NULL) AS did_search,
    (vw.user_id IS NOT NULL) AS did_view,
    (p.user_id IS NOT NULL) AS did_purchase
  FROM users u
  LEFT JOIN visits v ON v.user_id = u.user_id
  LEFT JOIN searches s ON s.user_id = u.user_id
  LEFT JOIN views vw ON vw.user_id = u.user_id
  LEFT JOIN purchases p ON p.user_id = u.user_id
)
SELECT
  acquisition_week,
  COUNT(*) FILTER (WHERE did_visit)    AS visits,
  COUNT(*) FILTER (WHERE did_search)   AS searches,
  COUNT(*) FILTER (WHERE did_view)     AS views,
  COUNT(*) FILTER (WHERE did_purchase) AS purchases,
  ROUND(COUNT(*) FILTER (WHERE did_search)::numeric   / NULLIF(COUNT(*) FILTER (WHERE did_visit),0), 4)  AS visit_to_search,
  ROUND(COUNT(*) FILTER (WHERE did_view)::numeric     / NULLIF(COUNT(*) FILTER (WHERE did_search),0), 4) AS search_to_view,
  ROUND(COUNT(*) FILTER (WHERE did_purchase)::numeric / NULLIF(COUNT(*) FILTER (WHERE did_view),0), 4)   AS view_to_purchase,
  ROUND(COUNT(*) FILTER (WHERE did_purchase)::numeric / NULLIF(COUNT(*) FILTER (WHERE did_visit),0), 4)  AS overall_conversion
FROM funnel
GROUP BY acquisition_week
ORDER BY acquisition_week;

Reproducible fixture and output (using a fixed as_of of 2026-09-01 00:00:00+00 in place of now(), so the demonstrated output is stable):

CREATE TABLE users (user_id text, created_at timestamptz);
CREATE TABLE events (user_id text, event_name text, "timestamp" timestamptz, properties jsonb);

INSERT INTO users VALUES
('u1','2026-08-03 00:00:00+00'),
('u2','2026-08-04 00:00:00+00'),
('u3','2026-08-10 00:00:00+00'),
('u4','2026-08-12 00:00:00+00');

INSERT INTO events VALUES
('u1','visit','2026-08-03 01:00:00+00','{}'),
('u1','search','2026-08-03 01:05:00+00','{}'),
('u1','view_product','2026-08-03 01:10:00+00','{}'),
('u1','purchase','2026-08-03 01:20:00+00','{}'),
('u1','visit','2026-08-04 01:00:00+00','{}'),
('u2','visit','2026-08-04 02:00:00+00','{}'),
('u2','search','2026-08-04 02:05:00+00','{}'),
('u2','view_product','2026-08-04 02:10:00+00','{}'),
('u3','visit','2026-08-10 03:00:00+00','{}'),
('u4','visit','2026-08-12 04:00:00+00','{}'),
('u4','view_product','2026-08-12 04:01:00+00','{}'),
('u4','search','2026-08-12 04:05:00+00','{}');

acquisition_week | visits | searches | views | purchases | visit_to_search | search_to_view | view_to_purchase | overall_conversion
2026-08-03        |   2    |    2     |   2   |     1     |     1.0000       |     1.0000      |      0.5000       |      0.5000
2026-08-10        |   2    |    1     |   0   |     0     |     0.5000       |     0.0000      |       null        |      0.0000

Key points

• The chained MIN(timestamp) joins are what "deduplicate events so each user counts once per funnel step in chronological order" means concretely: a user only advances past a step once they have a later qualifying event for the next step. u4 has a view_product event at 04:01, before their qualifying search at 04:05, so it does not count toward the view step; u4 correctly shows did_view = false even though a view_product event technically exists in the raw data.
• u1's duplicate visit event on Aug 4 is absorbed by MIN(timestamp), which takes the earliest visit (Aug 3) without inflating any count.
• The 30-day filter is applied to recent on the event's own timestamp, not on the user's acquisition date, so a long-tenured user active in the last 30 days is still included and correctly segmented by their own, possibly much older, acquisition week.
• COUNT(*) FILTER (WHERE ...) avoids a self-join or a CASE-heavy pivot to get per-step counts and step-to-step rates in one pass over funnel.

Complexity

Each step's CTE is an indexed range scan of recent joined back to the previous step's surviving users, so cost is proportional to the number of users who cleared the previous step, not the full event volume; the funnel narrows at each stage rather than re-scanning everything. Recommend a composite index on (user_id, event_name, timestamp) so both the per-step filter and the timestamp > ordering check are covered. The final aggregation is a single grouped pass, O(u) in the number of users (Big-O notation for how cost scales: work grows in direct proportion to the number of users, not to the total event volume).

Edge cases

• A user with a repeated event for the same step (u1's second visit): deduplicated by MIN(timestamp), no inflation.
• A user whose events are out of the funnel's logical order (u4's view before search): correctly excluded from the later step, demonstrated by views = 0 for that week even though a view_product row exists.
• A user with zero qualifying events in the window but an old acquisition date: still appears in funnel via the LEFT JOIN from users, contributing zero to every step without corrupting other cohorts' rates.
• A cohort with zero users at a given step: NULLIF renders the downstream ratio as null rather than dividing by zero or silently showing a misleading 0, as seen in view_to_purchase for the Aug 10 cohort.

Approach

Build two filtered views of the events table, one for signup events and one for purchase events, each truncated to a UTC calendar week. Left join purchases to signups matching on both user_id and the same week, so a user only counts as a converter if their purchase happened in the same calendar week as their signup, not merely at any point afterward. Aggregate with COUNT(DISTINCT ...) so a user with more than one event in a week is never double-counted.

SQL

WITH signups AS (
  SELECT
    user_id,
    date_trunc('week', event_time AT TIME ZONE 'UTC') AS signup_week
  FROM events
  WHERE event_name = 'signup'
),
purchases AS (
  SELECT
    user_id,
    date_trunc('week', event_time AT TIME ZONE 'UTC') AS purchase_week
  FROM events
  WHERE event_name = 'purchase'
)
SELECT
  s.signup_week AS week,
  COUNT(DISTINCT s.user_id) AS signups,
  COUNT(DISTINCT p.user_id) AS converters,
  ROUND(
    COUNT(DISTINCT p.user_id)::numeric / NULLIF(COUNT(DISTINCT s.user_id), 0),
    4
  ) AS conversion_rate
FROM signups s
LEFT JOIN purchases p
  ON p.user_id = s.user_id AND p.purchase_week = s.signup_week
GROUP BY s.signup_week
ORDER BY s.signup_week;

Reproducible fixture and output

CREATE TABLE events (user_id text, event_name text, event_time timestamptz, properties jsonb);
INSERT INTO events (user_id, event_name, event_time, properties) VALUES
('u1','signup','2026-08-03 10:00:00+00','{}'),
('u1','purchase','2026-08-05 10:00:00+00','{}'),
('u2','signup','2026-08-04 10:00:00+00','{}'),
('u2','purchase','2026-08-20 10:00:00+00','{}'),
('u3','signup','2026-08-05 10:00:00+00','{}'),
('u4','signup','2026-08-11 09:00:00+00','{}'),
('u4','purchase','2026-08-12 09:00:00+00','{}'),
('u5','signup','2026-08-12 09:00:00+00','{}'),
('u6','signup','2026-08-13 09:00:00+00','{}'),
('u6','purchase','2026-08-14 09:00:00+00','{}');

week          | signups | converters | conversion_rate
2026-08-03    |    3    |     1      |     0.3333
2026-08-10    |    3    |     2      |     0.6667

Week of Aug 3: u1 converts (purchase Aug 5, same week), u2 does not (purchase Aug 20, a different week), u3 never purchases. 1 of 3 signups converts, 0.3333. Week of Aug 10: u4 and u6 convert within the week, u5 never purchases. 2 of 3 converts, 0.6667.

Key points

• The join condition matches on (user_id, week), not just user_id, which is what enforces "at least one purchase in the week of signup" rather than "ever converts." u2 does eventually purchase, just two weeks later, and the query correctly excludes that from Aug 3's converter count.
• date_trunc('week', ...) truncates using the session's current timezone unless the timestamp is first shifted with AT TIME ZONE 'UTC'. Verified directly: a signup at 2026-08-03 03:00:00+00 (a UTC Monday) truncates to 2026-08-03 when converted to UTC first, but to 2026-07-27 (the wrong, prior week) under a Pacific-time session without that conversion, since 03:00 UTC is still Sunday evening in Pacific time.
• COUNT(DISTINCT ...) on both sides protects against a user firing more than one signup or purchase event in the same week from being counted more than once.

Complexity

Each CTE (a Common Table Expression, the named WITH signups AS (...) and WITH purchases AS (...) blocks defined above, which let you name and reuse an intermediate result instead of nesting subqueries) is a single filtered scan of events, O(n) in the number of matching rows (Big-O notation for how the work scales: O(n) means the time grows roughly in direct proportion to n, the number of matching rows, rather than, say, doubling faster than n grows) given an index on (event_name, user_id, event_time). The join between signups and purchases on (user_id, week) is a hash join (the database builds a fast in-memory lookup table from one side of the join, then checks each row from the other side against it, rather than comparing every row to every row), O(s + p) for s signups and p purchases, followed by an O(s) grouped aggregate. Recommend the composite index above so both the WHERE event_name = ... filters and the join key are covered without a full table scan.

Edge cases

• A user who signs up but never purchases: correctly stays in the signups denominator with no matching converter, via the LEFT JOIN (u3, u5 above).
• A user who eventually purchases, but in a later week than they signed up: correctly excluded from that week's converters, since the join requires the same week on both sides (u2 above); a naive query checking only "does this user_id appear anywhere in purchases" would over-count this case.
• A user with duplicate signup or purchase events in the same week: correctly counted once, via COUNT(DISTINCT user_id).
• Omitting AT TIME ZONE 'UTC': produces a wrong week boundary whenever the database session's timezone differs from UTC, as verified above; always pin the timezone explicitly rather than relying on the session default.

Direct answer

Low adoption is a diagnosis problem before it is a design problem. Instrument who has access, who has ever opened the dashboard, and who keeps returning, then segment those funnels by role so a healthy company-wide average can't hide a segment that never adopted at all. Layer qualitative interviews on top of the quantitative gap to find the "why," then run small, sequential A/B tests (a randomized comparison where users are split between a control and one or more variants) on messaging first and layout second, and track adoption with a small set of metrics that include usefulness, not just opens.

Structured elaboration

1. Product and usage data to collect (the adoption funnel)

   • Write every step with its denominator named, because "reach" means three different things if you do not, and a rate divided by the wrong population is not a weaker number, it is a different one. Fix the populations first: the intended audience is the count from role or license provisioning, and the openers are the subset who have opened the dashboard at least once.
   • Reach rate = users who have opened it at least once / intended audience. Answers "did it get in front of people at all."
   • Activation rate = users who took a meaningful action in their first session (filtered, exported, saved a view) / users who opened it at least once.
   • Return rate = weekly active users / users who have EVER opened it. Answers "do the people who tried it come back." This is the number most teams mean when they say retention, and its denominator is the openers, never the intended audience.
   • Weekly-active-reach = weekly active users / intended audience. This is the headline adoption number, and the two above multiply into it exactly: weekly-active-reach = reach rate x return rate. Reporting the headline without its two factors is the single easiest way to point a team at the wrong problem, because a low headline number looks identical whether nobody has heard of the dashboard or everybody bounced off it.
   • Depth: dwell time per panel and which sections get scrolled past unopened, which points at layout or relevance problems.
   • Cut every one of these by role, team, tenure, and access path (email link, nav, direct URL), because a blended number hides where the real drop-off lives.

2. Mixing qualitative feedback with quantitative signals

   • Use the quantitative funnel to find WHERE it breaks (a specific team sits far below the company average), then use a handful of short interviews or intercept surveys with people in that exact segment to find WHY: irrelevant default view, feels duplicative of an existing tool, too slow, or "I did not know it existed."
   • The two are not equally weighted at every stage: quantitative data tells you which segment and which funnel step to investigate; qualitative data adjudicates between several plausible explanations for that gap cheaply, before you spend engineering time building a fix around the wrong guess.

3. A/B tests to run

   • Messaging: in-product banners, onboarding tooltips, or an email nudge pointing the low-adoption segment to the dashboard, measured on click-through and 7-day return rate.
   • Layout: moving the most-requested panel above the fold, collapsing rarely-used ones, or changing the default view per segment, measured on activation rate and dwell time.
   • Run these sequentially rather than bundled: a messaging test reads out in days, a layout test needs longer to separate real signal from novelty.

4. Success metrics for increased adoption

   • Primary: weekly-active-reach (weekly active users divided by intended audience), tracked per segment, not only in aggregate, and always reported next to its two factors (reach rate and return rate) so a flat headline cannot hide one factor improving while the other decays.
   • Guardrails: task completion rate on the dashboard's core job, and time-to-first-insight (time from open to first meaningful interaction), so opens are not rising while usefulness stays flat.

Worked example

Say the dashboard has 850 intended users from role provisioning. Usage logs show 120 have opened it at least once in the last 90 days and 45 are weekly active. Those two counts give three different rates, each on its own stated denominator:

• reach rate = 120 / 850 = 14.1%
• return rate = 45 / 120 = 37.5%
• weekly-active-reach = 45 / 850 = 5.3%, and the identity holds: 14.1% x 37.5% = 5.3%

Read the factors, not just the product. A 37.5% return rate is unremarkable for an internal tool; the 14.1% reach rate is where the product is actually failing. A report showing only the blended 5.3% would have sent the team off to redesign the dashboard when most of the intended audience has never opened it once.

Segmenting by team, with both factors carried through:

That gap, not the blended 5.3%, is the real target, and the split denominators say something the single number cannot: Ops is failing on BOTH factors, a 10% reach rate (nine in ten Ops users have never opened it, an awareness or routing problem) and a 20% return rate among the few who did (a relevance problem). Five short interviews with Ops users surface that the default view opens on revenue metrics Ops does not own, so they bounce immediately. That explains the return rate and says nothing about the reach rate, which needs its own intervention. So the layout A/B test comparing a team-aware default view (Ops sees operational metrics first) against the current one-size-fits-all default should read out on Ops RETURN rate, the factor it can plausibly move, with Ops time-to-first-insight as the supporting diagnostic. Reading it out on Ops weekly-active-reach would be a mistake with a predictable cost: even a fix that doubled the Ops return rate from 20% to 40% would only move Ops weekly-active-reach from 2% to 4%, because it is diluted by the 90% of Ops the layout change never reaches, and a 2-point move on a 2% base is easy to dismiss as noise.

Trade-offs and pitfalls

A single company-wide adoption number rewards fixes aimed at the segment that had already adopted, since that is the segment moving the average. Running the messaging and layout tests on the same population at the same time confounds which lever actually moved the number, so sequence them or split the population cleanly. And a spike in opens right after a messaging nudge is not adoption by itself: count it as progress only if activation or retention also moves, otherwise it measured curiosity rather than a habit forming.