Design Metrics and Impact Measurement Questions

Direct answer

Converting each step: Visited to Started Trial is 12%, Started Trial to Completed Onboarding is 50%, Completed Onboarding to First Paid Conversion is 10%, and overall Visited to Paid is 0.6%. The single biggest opportunity is the Started Trial to Completed Onboarding step: half of everyone who deliberately started a trial, meaning they already cleared the harder top-of-funnel decision, never finishes onboarding. Unlike the other two drops, that step sits squarely inside the design and product team's control, rather than being shaped mostly by marketing quality (top of funnel) or pricing and billing (paid conversion).

Structured elaboration

Why this stage over the other two candidates:

• Visited to Started Trial (12%) has the largest absolute headcount loss, 8,800 people, but top-of-funnel conversion for a cold visitor is naturally low across most SaaS products and is heavily shaped by marketing and positioning outside the design team's direct control.
• Completed Onboarding to Paid (10%) has the lowest single-stage rate, but for someone who already finished onboarding, non-conversion is often a pricing, billing-flow, or business-model question as much as a design one.
• Started Trial to Completed Onboarding (50%) means half of the people who already made the harder commitment of starting a trial never reach the point where the product can prove its value at all. That is a pure activation and onboarding-design problem: friction, unclear steps, or a setup that asks for too much before showing value.

Worked example

If Started Trial to Completed Onboarding improves from 50% to 65%, a realistic target for removing friction such as cutting required setup fields, and the downstream 10% Onboarded-to-Paid rate holds constant:

That is 78 paid customers versus the current 60, a 30% relative increase, entirely from fixing one mid-funnel step.

Now be careful about the comparison most candidates reach for next. Pushing the top of funnel from 12% to 15% gives:

a 25% increase, which looks smaller than the mid-funnel result. It is smaller only because it is a smaller push. 50% to 65% is a 30% relative lift (65/50 = 1.30); 12% to 15% is a 25% relative lift (15/12 = 1.25). The two are not the same size of ask, so the comparison as stated does not support "mid-funnel is the better lever."

Put them on the same basis and the illusion disappears. Take the same 30% relative lift at the top instead, 12% to 15.6%:

Exactly 78, identical to the mid-funnel result. That is not a coincidence: the funnel is a product of three rates, so multiplying any one of them by 1.30 multiplies the final paid count by 1.30. An x% relative lift is worth the same number of paid customers no matter which stage it happens at.

The real argument for the mid-funnel step is therefore not multiplicative leverage, it is achievability and ownership: a 30% relative lift on trial-to-onboarded (50% to 65%) is a plausible outcome of removing setup friction the design team controls, whereas a 30% relative lift on cold-visitor-to-trial (12% to 15.6%) requires moving positioning, pricing and traffic mix that mostly sit outside the team. Argue for the stage on the size of lift you can credibly deliver there, not on where the lift sits in the funnel.

Two experiments targeting the Started Trial to Completed Onboarding gap:

1. Cut the onboarding flow to a minimum viable setup, deferring optional fields and integrations until after first value is shown, measured by onboarding completion rate and time-to-first-value.
2. Add a visible progress indicator plus one quick-win step early in onboarding, such as importing one data source and seeing one result, before asking for anything else, measured by completion rate and drop-off point by step.

Trade-offs & pitfalls

Do not assume the multiplicative math means all three stages are interchangeable levers, and do not make the opposite mistake either. Because the funnel multiplies, equal relative lifts at different stages are exactly equal in paid customers, so a comparison that appears to favour one stage is almost always comparing two different-sized lifts. Each stage has a different owner and a different achievable relative lift, so the only thing that separates the stages is which one you can credibly move by more, not where it sits.

Also keep percentage points and relative percent apart when you argue this. Trial-to-onboarded moving from 50% to 65% is +15 percentage points and +30% relative; visited-to-trial moving from 12% to 15% is only +3 percentage points but +25% relative. The percentage-point figures are not comparable across stages with different baselines; the relative figures are. Also watch for a common mistake in framing this kind of problem: treating the lowest single-stage percentage (Onboarded to Paid at 10%) as automatically the biggest opportunity, when the real question is which gap the team can both move meaningfully and control directly.

Approach

Build two filtered views of the events table, one for signup events and one for purchase events, each truncated to a UTC calendar week. Left join purchases to signups matching on both user_id and the same week, so a user only counts as a converter if their purchase happened in the same calendar week as their signup, not merely at any point afterward. Aggregate with COUNT(DISTINCT ...) so a user with more than one event in a week is never double-counted.

SQL

WITH signups AS (
  SELECT
    user_id,
    date_trunc('week', event_time AT TIME ZONE 'UTC') AS signup_week
  FROM events
  WHERE event_name = 'signup'
),
purchases AS (
  SELECT
    user_id,
    date_trunc('week', event_time AT TIME ZONE 'UTC') AS purchase_week
  FROM events
  WHERE event_name = 'purchase'
)
SELECT
  s.signup_week AS week,
  COUNT(DISTINCT s.user_id) AS signups,
  COUNT(DISTINCT p.user_id) AS converters,
  ROUND(
    COUNT(DISTINCT p.user_id)::numeric / NULLIF(COUNT(DISTINCT s.user_id), 0),
    4
  ) AS conversion_rate
FROM signups s
LEFT JOIN purchases p
  ON p.user_id = s.user_id AND p.purchase_week = s.signup_week
GROUP BY s.signup_week
ORDER BY s.signup_week;

Reproducible fixture and output

CREATE TABLE events (user_id text, event_name text, event_time timestamptz, properties jsonb);
INSERT INTO events (user_id, event_name, event_time, properties) VALUES
('u1','signup','2026-08-03 10:00:00+00','{}'),
('u1','purchase','2026-08-05 10:00:00+00','{}'),
('u2','signup','2026-08-04 10:00:00+00','{}'),
('u2','purchase','2026-08-20 10:00:00+00','{}'),
('u3','signup','2026-08-05 10:00:00+00','{}'),
('u4','signup','2026-08-11 09:00:00+00','{}'),
('u4','purchase','2026-08-12 09:00:00+00','{}'),
('u5','signup','2026-08-12 09:00:00+00','{}'),
('u6','signup','2026-08-13 09:00:00+00','{}'),
('u6','purchase','2026-08-14 09:00:00+00','{}');

week          | signups | converters | conversion_rate
2026-08-03    |    3    |     1      |     0.3333
2026-08-10    |    3    |     2      |     0.6667

Week of Aug 3: u1 converts (purchase Aug 5, same week), u2 does not (purchase Aug 20, a different week), u3 never purchases. 1 of 3 signups converts, 0.3333. Week of Aug 10: u4 and u6 convert within the week, u5 never purchases. 2 of 3 converts, 0.6667.

Key points

• The join condition matches on (user_id, week), not just user_id, which is what enforces "at least one purchase in the week of signup" rather than "ever converts." u2 does eventually purchase, just two weeks later, and the query correctly excludes that from Aug 3's converter count.
• date_trunc('week', ...) truncates using the session's current timezone unless the timestamp is first shifted with AT TIME ZONE 'UTC'. Verified directly: a signup at 2026-08-03 03:00:00+00 (a UTC Monday) truncates to 2026-08-03 when converted to UTC first, but to 2026-07-27 (the wrong, prior week) under a Pacific-time session without that conversion, since 03:00 UTC is still Sunday evening in Pacific time.
• COUNT(DISTINCT ...) on both sides protects against a user firing more than one signup or purchase event in the same week from being counted more than once.

Complexity

Each CTE (a Common Table Expression, the named WITH signups AS (...) and WITH purchases AS (...) blocks defined above, which let you name and reuse an intermediate result instead of nesting subqueries) is a single filtered scan of events, O(n) in the number of matching rows (Big-O notation for how the work scales: O(n) means the time grows roughly in direct proportion to n, the number of matching rows, rather than, say, doubling faster than n grows) given an index on (event_name, user_id, event_time). The join between signups and purchases on (user_id, week) is a hash join (the database builds a fast in-memory lookup table from one side of the join, then checks each row from the other side against it, rather than comparing every row to every row), O(s + p) for s signups and p purchases, followed by an O(s) grouped aggregate. Recommend the composite index above so both the WHERE event_name = ... filters and the join key are covered without a full table scan.

Edge cases

• A user who signs up but never purchases: correctly stays in the signups denominator with no matching converter, via the LEFT JOIN (u3, u5 above).
• A user who eventually purchases, but in a later week than they signed up: correctly excluded from that week's converters, since the join requires the same week on both sides (u2 above); a naive query checking only "does this user_id appear anywhere in purchases" would over-count this case.
• A user with duplicate signup or purchase events in the same week: correctly counted once, via COUNT(DISTINCT user_id).
• Omitting AT TIME ZONE 'UTC': produces a wrong week boundary whenever the database session's timezone differs from UTC, as verified above; always pin the timezone explicitly rather than relying on the session default.

Direct answer

Pick one North Star Metric (the single metric a team believes best captures the core value delivered
to customers, tightly linked to long-term business health) for the subscription product, break it into
four leading indicators that design and product teams can realistically move within a quarter and that
predict the North Star moving next, and set two guardrail metrics that must never be allowed to quietly
regress while the team optimizes the leading indicators.

Structured elaboration

• Why this North Star. "Weekly active subscribers completing their core job" is preferred over raw
  Monthly Recurring Revenue (MRR) here because MRR lags team actions by weeks or months and can move
  from pricing or sales activity independent of product quality, while successful weekly usage sits
  much closer to the levers design and product teams actually control, and is still causally upstream
  of retention and revenue.
• How each level informs tactical decisions. The North Star sets the multi-quarter direction that
  leadership tracks and rarely moves week to week. The leading indicators are what individual teams
  pick as their quarterly OKR (Objectives and Key Results) Key Results, since each is a metric a
  specific roadmap bet can realistically move within a quarter. The guardrails are checked as a gate at
  every launch decision, not optimized directly, the same role a guardrail metric plays in a
  controlled experiment: you don't try to improve it, you make sure nothing you did made it worse.
• Presenting visually to executives. A single-slide top-down layout: the North Star in one large
  box at the top, the four leading indicators in a row beneath it, each with a one-line caption stating
  why it predicts the North Star, and the two guardrails set visually apart, a different color, off to
  the side, labeled "must not move backward" so the room can't mistake a guardrail for something to
  maximize.

Worked example

Suppose the design team ships a redesigned first-run tutorial targeting leading indicator #1, day-7
activation rate. Before shipping, the team defines success with real numbers: day-7 activation rate has a pre-launch baseline of 38%, and a pre-registered minimum detectable effect (MDE, the smallest lift worth the engineering and design cost of keeping the change) of +3 percentage points, so the bar for success is 41% or higher. The two guardrails get explicit bounds too: involuntary churn (baseline 2.1%) must stay within 0.2 points of that baseline, and complaint rate (baseline 1.4%) must stay within 0.3 points of its baseline. After the test runs, activation rate lands at 42.6%, a +4.6 point lift that clears the +3 point MDE; involuntary churn holds at 2.0%, inside its guardrail band; and complaint rate holds at 1.5%, also inside its guardrail band. Only with all three numbers in hand, not activation moving alone, does the team credit that quarter's leading-indicator movement toward the North Star, rather than declaring victory on activation alone while a guardrail quietly slipped.

Trade-offs and pitfalls

A North Star chosen for how good it looks in a board deck rather than for causal proximity to what
teams actually do demotivates teams, since no one can move it directly. More than four or five leading
indicators dilutes focus and recreates the "everything is a priority" problem the hierarchy was meant
to solve. And guardrails that are never actually enforced, meaning no launch is ever blocked over them,
become theater rather than a real check.

Direct answer

The one-pager needs to work as both a launch narrative and a living reference: a headline result stated in plain business terms first, the primary metric with its size of effect and how confident we are in it, one or two supporting metrics, a guardrail check confirming nothing else broke, and a clear recommendation for what happens next. I'd design it as a dashboard-style structure that keeps getting reused as the launch matures, not a one-time slide that gets forgotten the week after it's presented.

Structured elaboration

Elements to include:

• A headline banner in business language ("Checkout redesign lifted revenue-driving conversions, no negative impact on support volume") rather than a metric name and a percentage as the first thing anyone reads.
• The primary metric shown as a before/after comparison with the size of the change and a plain-language confidence statement, not just a bare percentage.
• One or two supporting metrics that corroborate the primary result (for a conversion lift, something like time-to-checkout or drop-off rate at the step that changed).
• A guardrail line confirming the metrics that shouldn't move, didn't move badly (error rate, support tickets, adjacent-flow performance).
• A small trend chart showing the metric over the surrounding weeks, so the audience can see the change wasn't a blip before or after the launch window.
• A short qualitative note or representative user quote, since a number alone rarely persuades a non-design audience as effectively as a number paired with a human reason.
• A clearly labeled recommendation and next step (ship to 100%, iterate further, hold and investigate).

Communicating uncertainty: state the confidence interval and the sample size or duration behind it in plain language rather than showing a raw statistical output, for example "we're 95% confident the true lift is between 0.08 and 0.52 percentage points, based on 100,000 sessions over two weeks" rather than just "z = 2.68, p < 0.01." If the interval is wide, say so plainly rather than letting the headline point estimate imply more precision than the data supports.

Structuring it as an ongoing dashboard, not just a launch snapshot: the same layout, headline banner, metric tiles, trend chart, guardrail row, doubles as a recurring reference stakeholders can revisit weeks later to see whether the initial win held up, rather than a static slide that gets forgotten. Treating the one-pager as a durable dashboard rather than a one-time announcement is what makes the "successful launch" story stay true past the first week, since a redesign that looks great on day 3 but decays by day 30 is a very different result than one that holds.

Worked example

Checkout redesign, two-week window, 50,000 sessions per arm: baseline conversion 3.10% (1,550 conversions), post-change conversion 3.40% (1,700 conversions).

The lift and its interval both have to be carried on the same basis, percentage points, or the arithmetic silently stops meaning anything:

That 0.00112 is a proportion, which is 0.112 percentage points; converting it before it meets the 0.30 is the whole trick.

That is a 0.30 percentage-point absolute lift and, separately, a relative lift of about 9.7% (0.30 divided by 3.10). Those are two different bases and the page has to label which one it is quoting: 9.7 percentage points would be roughly thirty-two times the actual 0.30-point lift, and an exec who reads the relative number as an absolute one has misjudged the result by that factor. The confidence interval doesn't cross zero (the same standard error gives $z = 0.0030 / 0.00112 \approx 2.68$), so the headline can honestly read "conversion lift, 95% confident the true improvement is between roughly 0.1 and 0.5 percentage points" rather than a single overconfident number.

Trade-offs and pitfalls

The biggest trap is choosing metrics for the one-pager that flatter the result rather than test it; if the guardrail row is quietly dropped because nothing interesting happened there, a skeptical stakeholder should assume the omission was deliberate, not that everything was fine. The second trap is presenting a point estimate with no interval at all, which reads as more certain than the underlying data justifies and sets up an awkward conversation later if the effect doesn't hold at full rollout.

Direct answer

In the first 72 hours, the dashboard's job is fast detection of breakage, not deep insight. Pick one primary metric that's a direct proxy for "is the core job still getting done," wrap it in a small number of guardrails that catch failure modes the primary metric would be slow to reflect, keep data fresh enough to react within the window, and route each alert to whoever can actually act on it. Only after the acute window closes does the view widen into a longer adoption-and-health dashboard.

Structured elaboration

• Primary metric: report generation success rate, successful completions divided by generation attempts, tracked per rolling window.
• Guardrail metrics: p95 report generation latency (the 95th-percentile generation time: 95% of reports finish faster than this value and the slowest 5% take longer, catches "still succeeds, but painfully slow"), backend error rate on the report API (a leading indicator that trips before user-visible failures pile up), and export or download completion rate (catches breakage downstream of generation itself, such as a broken export button).
• Data freshness: at 20,000 reports a day, that's roughly 14 reports a minute on average, enough volume for a 15-minute rolling window to give a stable percentage. For the highest-risk first 6 hours after launch, tighten to a 5-minute window to catch a bad deploy as fast as possible, then relax back to 15 minutes for the remainder of the 72 hours.
• Alert thresholds and routing:

The routing split is a deliberate design decision, not a default: the two backend-facing alerts page engineering because only engineering can act on them fast; a latency dip alone rarely means total failure, so it gets visibility without a page; and an export-completion drop may point to a UX confusion issue design needs to see, not necessarily a backend outage.

Once the 72-hour acute window closes, fold this into a longer-running 12-week adoption and health view: one top-line "reporting health" number blending success rate and latency that any stakeholder can glance at, with drill-downs underneath by report type, customer segment, and device, so a product manager asking "is this working" gets the headline, and an engineer asking "for whom is it not working" can click into the same dashboard to find the segment.

Worked example

At the stated scale of 500,000 monthly active users and 20,000 reports generated a day, be careful with the obvious 20,000 / 500,000 = 4% division: the numerator counts daily REPORTS and the denominator counts monthly USERS, so the quotient is neither a penetration rate nor a daily-active share, and it is only an upper bound on how many distinct users touch reporting in a day. Converting it needs one more input, reports per reporting user per day: at about 1.3, roughly 20,000 / 1.3 = 15,400 distinct users touch reporting on a given day, about 15,400 / 500,000 = 3% of MAU. Label whichever basis the tile is showing, because "3% of our users use reports" and "reports run at 4% of MAU per day" get read as the same claim and are not. On either basis the feature is a meaningful but minority slice of overall usage, so a total outage here should be clearly visible but won't dominate whole-product metrics on its own. Average report volume is $20{,}000 / 1{,}440 \approx 13.9$ reports per minute, so even a 5-minute window carries roughly 70 reports, enough to compute a meaningful success percentage rather than reacting to single-digit noise. If the historical baseline success rate is 99.5%, the 97% paging threshold represents a real 2.5-point drop, sustained across two windows so a single noisy blip doesn't trigger a page.

Trade-offs and pitfalls

Paging on too many metrics at launch trains the on-call engineer to start ignoring alerts, cap paging metrics at 2 to 3 and keep everything else visible but non-paging. Tighter windows are noisier with lower counts per window, which is why the early 5-minute window keeps a wider absolute threshold band rather than the same tight one used later at 15 minutes. And a single global success-rate threshold can mask one report type failing completely while others compensate in the blended average, a senior plan should flag that risk and consider per-report-type thresholds once volume allows it, rather than stopping at one aggregate number.