Design Metrics and Impact Measurement Questions

Answer (Product Designer perspective)

Assumptions

• Monthly active users (MAU): 200,000
• Baseline checkout conversion: 8% (0.08)
• Target improvement: +2 percentage points → 10% (0.10)
• Average order value (AOV): $80
• Gross margin: 60% (0.60)
• One-time redesign implementation cost: $120,000
• No change in traffic or AOV beyond conversion lift

Formula — incremental orders & revenue

Incremental orders per month = MAU * (new_conv - old_conv)

Plain English: extra customers who complete checkout each month.

Incremental gross profit per month = Incremental orders * AOV * gross_margin

Numbers

• Incremental orders = 200,000 * (0.10 - 0.08) = 4,000 orders/month
• Incremental gross profit = 4,000 * $80 * 0.60 = $192,000/month

ROI formula

ROI (%) = (Total incremental gross profit over period - implementation_cost) / implementation_cost * 100

If measured over 12 months:

• Annual incremental gross = $192,000 * 12 = $2,304,000
• ROI = (2,304,000 - 120,000) / 120,000 * 100 ≈ 1,820%

Payback period

Payback months = implementation_cost / incremental_gross_profit_per_month

= 120,000 / 192,000 ≈ 0.625 months (~19 days)

Notes / Sensitivity

• As a designer I’d validate assumptions via A/B test and consider conservative scenarios (e.g., 1 pp lift). Also factor implementation vs. recurring costs, and impact on LTV (repeat purchases) for long-term value.

Overview & Hypotheses
I would run a rigorous evaluation to demonstrate causality between the redesign and revenue/retention. Primary hypotheses:

• H1: The redesign increases 28-day retention rate by at least Δ1 (e.g., +3 percentage points).
• H2: The redesign increases 90-day revenue per user (RPU) by at least Δ2 (e.g., +5%).

Experimental design

• Preferred: Randomized A/B test at user-level (or device-level if logged-out). Randomly assign users to treatment (redesign) or control (current UI).
• If rollout constraints exist: use a stepped-wedge or difference-in-differences (DiD) quasi-experiment across regions/segments with staggered launch.

Metrics

• Primary metrics: 28-day retention (binary), 90-day revenue per user (continuous).
• Secondary metrics: engagement (session count, time on task), conversion funnel rates, NPS/qualitative satisfaction.
• Guardrail metrics: error rates, load time, support contacts.

Sampling & power

• Precompute sample size from baseline rates and minimum detectable effect (MDE). Example: for retention baseline 20%, MDE 3pp, α=0.05, power=0.8 → use standard proportion power formula to get N per arm. For revenue (skewed), use log-transform or bootstrap for power estimation.

Controls & confounders

• Collect covariates: user tenure, device/OS, acquisition channel, geography, prior engagement, subscription tier. Use stratified randomization or block on major covariates. In DiD, control for time trends and cohort fixed effects.

Analysis & statistical tests

• Primary: Intention-to-treat (ITT) estimates.
• Retention: logistic regression or linear probability model with covariates; report marginal effects and 95% CIs.
• Revenue: if skewed, use log(RPU+1) OLS or quantile regression; supplement with nonparametric bootstrap.
• For quasi-experiment: DiD regression with time and cohort fixed effects, test parallel trends pre-check.
• Multiple comparisons: adjust p-values (Holm/BH) if testing many metrics.
• Sensitivity: subgroup analyses, per-protocol, and instrumental variable (if noncompliance).

Interpretation & UX context

• Combine quantitative effect sizes with qualitative feedback (task success, usability tests) to show mechanism (why revenue/retention rose). Present lift, CI, and business impact (projected revenue) to stakeholders to secure budget.

Overview / Goal
Demonstrate that keyboard-only and screen‑reader improvements increased task success, efficiency, and perceived accessibility for assistive‑tech users.

Quantitative KPIs & Instrumentation

• KPIs:
  • Task success rate (completion) for keyboard-only & screen‑reader users
  • Time-on-task (median) for those groups
  • Error rate (keyboard navigation errors; aria state mismatches)
  • Drop-off at critical flows (percent abandon at form steps)
  • Usage of accessibility affordances (skip links clicked, focus-visible events)
  • Customer-reported accessibility issues / support tickets
• Events to instrument (production analytics):
  • a11y_session_start { user_id_opt, assistive_tech: keyboard|screenreader|none }
  • focus_move { element_id, method: keyboard|mouse, timestamp }
  • skiplink_click { page, timestamp }
  • form_field_error_focus { field_id, error_type }
  • task_complete { task_id, success: true/false, duration_ms }
  • a11y_opt_in / detected_attribution (browser AT heuristics)
• Tag events with user cohort (self‑reported AT in profile / detected heuristics) and anon hashed id for longitudinal analysis.

Recruitment & Assisted‑Tech Usability Sessions

• Recruit 8–12 participants per AT cohort (keyboard-only; NVDA/VoiceOver users), prioritizing real-world users and diversity of tasks.
• Screening: frequency of AT use, proficiency, device/OS, primary tasks.
• Test design:
  • Remote moderated sessions using user's preferred setup; provide pre-test checklist.
  • 4–6 realistic tasks mapped to KPI flows (e.g., find/save, complete form, navigate modal).
  • Think‑aloud encouraged, with a short warm-up keyboard navigation task.
  • Capture screen + mic, keystroke logs, and screen‑reader audio.
  • Post-task SUS + bespoke accessibility satisfaction rating and qualitative probes.

Analyses for Small Samples

• Descriptive: per-user task success, median time, error counts; violin/box plots per cohort.
• Nonparametric tests: Wilcoxon signed‑rank for pre/post paired tasks; Mann‑Whitney U for independent groups.
• Use exact binomial CI for task success rates (report 95% CI).
• Combine quantitative with qualitative clustering: map recurring issues to severity and frequency.
• Triangulate production telemetry with lab findings (e.g., spikes in abandon ≈ observed failure points).

Reporting Limitations

• Small sample limits generalizability; report CIs and effect sizes not just p‑values.
• Detection of AT in production is imprecise — declare heuristic accuracy and reliance on self‑report.
• Confounds: user proficiency, device/OS variance, feature exposure timing.
• Recommend ongoing monitoring, A/B rollout when feasible, and follow‑up larger cohorts for confirmation.

This plan ties measurable production signals to targeted qualitative sessions so stakeholders see both numeric impact and the user stories behind them.

Direct answer / required inputs

• Inputs: baseline conversion p1 = 0.05, minimum detectable effect (MDE) = 3% relative → absolute delta = 0.05 * 0.03 = 0.0015, significance α = 0.05 (two-sided), power 1−β = 0.80. Also need allocation (typically 50/50) and expected loss-to-follow-up or QA exclusions.

Sample-size estimate (approx.)

• Using the two-proportion normal approximation:

n_per_group ≈ ( (Z_{α/2} * sqrt(2 * p_avg * (1 - p_avg))
               + Z_{β} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / delta^2

• Plugging p1=0.05, p2=0.0515, Z_{α/2}=1.96, Z_{β}=0.84 gives ~336,000 users per arm → ~672,000 total.

Why it’s so large

• A 3% relative uplift on a small base rate (5%) is an extremely small absolute change (0.15 percentage points). Smaller absolute deltas require quadratically more samples.

Trade-offs: power vs duration

• Increasing power (e.g., 90%) raises required N and therefore experiment duration or traffic diverted.
• Shorter duration (fewer users) reduces power -> higher false-negative risk.
• Alternatives: increase MDE (test bigger changes), target higher-conversion segments, run longer, or use sequential methods / Bayesian approaches (with proper controls).

Practical steps for a product designer

• Align with analytics: confirm baseline, consistent event definitions, and expected funnel drop-offs.
• Choose MDE by business impact (what uplift justifies rollout?) not only statistical nicety.
• Run power calculations together (consider clustering, seasonality, multiplicity).
• Consider targeting or phasing: test on high-traffic segments, pilot with stronger variations, or use commitment to longer test windows.
• Pre-register metrics and stopping rules, and ensure instrumentation and QA before ramping.

Approach (high-level)
I would run a causal, multi-pronged analysis to separate selection effects (the audience changed) from a design-driven uplift (UX/product change improved conversion). I’d combine segmentation, causal models, and sanity checks.

1. Clarify window & baseline

• Define pre/post windows around the external event. Exclude transition noise. Pick comparable weeks.

2. Key segments to compare

• Traffic source (paid/organic/referral), campaign, landing page, geo, device, new vs returning, cohort by acquisition date, time of day.

3. Descriptive checks first

• Plot top-of-funnel volume and conversion rate by segment over time. Look for segments where volume dropped most and conversion jumped.

4. Causal tests / models

• Difference-in-differences: compare conversion changes in affected segments vs unaffected controls (e.g., unaffected geos or channels).
• Interrupted time series: test abrupt change coincident with event controlling for trend/seasonality.
• Logistic regression with covariates (source, device, session duration, landing page) and an indicator for post-period; check interaction terms.
• Propensity score matching: match pre/post users on observables and compare conversion.

5. Design-driven signals to look for

• Lift concentrated on specific variants/landing pages, A/B test flags, UI changes deployed at same time.
• Behavioral signals: lower dropoff times, higher engagement on funnels, heatmap/scroll metrics improved.

6. Selection-driven signals to look for

• Remaining users have higher historical LTV, more direct/organic intent, shorter path-to-convert, different device/geo mix.

7. Statistical tests & metrics

• Two-sample proportion tests (z-test) per segment, t-tests for engagement metrics, Wald tests for regression coefficients. Adjust p-values for multiple comparisons.

8. Sanity checks

• Verify instrumentation and attribution, check for bot traffic, confirm no concurrent marketing spend changes, replicate in different time windows, and validate with qualitative research (user interviews/session replays).

Outcome: combine quantitative causality evidence + qualitative insights. If models show lift concentrated on unchanged pages but on higher-intent segments → selection. If lift persists after matching and concentrated where UX changed → design-driven.