Situation: We want to test two dashboard onboarding flows (A = current, B = new) to see if B increases users who complete onboarding and engage with the dashboard.Primary metric:- Onboarding conversion: proportion of new users who complete the onboarding flow and view at least one core dashboard within 7 days (binary per user).Secondary metrics:- Time-to-first-dashboard (median hours)- 7-day retention (return to dashboard >=1 time)- Downstream engagement (avg dashboards viewed per user in 14 days)- Error/abandonment rate during onboardingRandomization & assignment:- Randomize at user-id level at first app/web session after sign-up to avoid cross-over.- Use deterministic hashing (e.g., hash(user_id) mod 10000 < 5000 → variant A; else B) so assignment is persistent across sessions and platforms.- Exclude internal/test accounts and users with previous onboarding history.Instrumentation:- Track events: onboarding_started, onboarding_completed, dashboard_view (include user_id, timestamp, platform, experiment_id, variant).- Example SQL to compute primary metric:sql
WITH users AS (
SELECT user_id, variant
FROM experiment_assignments
WHERE experiment_id = 'onboard_v2' AND assigned_at <= CURRENT_DATE
),
converted AS (
SELECT u.user_id
FROM users u
JOIN events e ON e.user_id = u.user_id
WHERE e.event_name = 'dashboard_view'
AND e.timestamp BETWEEN u.assigned_at AND u.assigned_at + INTERVAL '7 days'
)
SELECT variant,
COUNT(DISTINCT user_id) AS total_users,
COUNT(DISTINCT converted.user_id) AS converters,
COUNT(DISTINCT converted.user_id)::float / COUNT(DISTINCT user_id) AS conversion_rate
FROM users
LEFT JOIN converted ON users.user_id = converted.user_id
GROUP BY variant;
