Design Metrics and Impact Measurement Questions

Analysis Plan — what I'll measure and why

• Define cohorts: New users (first 30 days), power users (top 10% by frequency/feature use), and intermediate.
• Pull event-level data for 30 days pre/post redesign: conversion funnel steps, time-on-task, feature usage, error/exit rates, screen-by-screen dropoff.
• Segment by behavioral (session frequency, feature affinity, task flows used) and demographic (device, OS, locale, age cohort if available).
• Use delta analysis + statistical testing (chi-square or proportion z-tests) to find statistically significant changes per segment.
• Visuals: funnel comparison heatmaps, cohort retention curves, and session replay sampling for impacted segments.

Findings interpretation checklist

• Is drop concentrated on a specific task/flow or device/viewport?
• Did visual changes hide or deprioritize power-user shortcuts (keyboard, right-click, advanced filters)?
• Are microcopy and affordances confusing for experienced patterns?

Product/UI mitigations (by segment)

• Power users (high-frequency, feature-heavy)
  • Reintroduce or surface power-user shortcuts: keyboard shortcuts, long-press menus, quick-actions.
  • Offer a “Compact/Advanced” UI toggle that preserves streamlined new-user layout but restores dense controls.
  • Improve discoverability of expert features with contextual affordances (tooltips that appear once per user).
• Intermediate users
  • Progressive disclosure: show simplified controls by default, reveal advanced options inline when used.
  • Add persistent contextual help that nudges toward advanced features.
• New users (benefited)
  • Preserve simplified flows and onboarding that improved conversion.
  • Add optional “learn more” affordance to avoid regressions.

Validation / measurement plan

• Run A/B tests: control (current redesign) vs variant(s) with mitigations (e.g., advanced toggle, restored shortcuts).
• Primary metrics: power-user conversion rate, time-to-convert for power users, retention of power users at 7/30 days.
• Secondary: new-user conversion (ensure no regression), task completion time, error rates, NPS for experienced users.
• Success criteria: statistically significant restoration of power-user conversion to within X% of baseline without reducing new-user lift; monitor via sequential testing and stop-loss rules.
• Qualitative: session replays and targeted usability tests with power users to validate friction removed.

As a UI designer I’d prototype the toggles/shortcuts in Figma, validate with quick moderated tests, then hand off to experiment engineers with annotated specs and accessibility considerations.

Hypothesis
Tighter spacing increases information density and scannability, improving task efficiency (faster time-to-target) without reducing comprehension. Looser spacing improves perceived clarity and preference but may increase scroll/time.

A/B test design

• Variants: Tight spacing vs Loose spacing (identical content/typography)
• Primary metric: Task completion time (for a defined microtask) or conversion/CTA click-through if applicable
• Secondary metrics: Task success rate, abandonment rate, bounce, Dwell time, micro-interactions (hover/click), SUS-like satisfaction score post-task

Sample size considerations

• Set α = 0.05, power = 0.8, choose a minimum detectable effect (MDE) — e.g., detect 5% change in CTR or 10% change in mean task time.
• Use a sample size calculator (or z-test formula). Example: baseline CTR 20%, MDE 5pp → ~1,400–1,800 users per variant. If measuring time, estimate SD from pilot and compute n with t-test formula. Run for one full business cycle and check for novelty/segment balance.

Complementary qualitative usability test

• Method: Moderated think-aloud sessions (6–8 per variant) + task-based scenarios reflecting A/B tasks. Use remote recording.
• Measures: Observed friction points, reading path, misclick causes, perceived hierarchy, cognitive load, preference interview (A/B preference and why). Capture screenshots, heatmaps, and short follow-up survey (Likert clarity/readability).

What you’ll learn / how to act

• Quantitative: whether spacing affects efficiency or conversion and magnitude.
• Qualitative: why differences exist — e.g., tight layout causes visual crowding or helps scanning; loose layout breaks groupings. Use insights to iterate (adjust spacing rhythm, increase grouping, adjust line-height) and target segments (mobile vs desktop).

Overview (goal)
Design a clear, scannable dashboard to track adoption and health of "Saved Items" over first 12 weeks—support product/UX decisions and surface issues quickly for designers and PMs.

Primary KPIs

• Weekly Active Users with Saved Items (WAU-SI) — adoption velocity
• % of new users who save at least once within 7 days — activation rate
• Retention of savers at week 1/4/12 — stickiness
• Avg saves per active saver — engagement

Supporting metrics

• Total saves, unique savers, saves per session, save failure rate, time-to-first-save, removal/un-save rate, per-platform breakdown (iOS/Android/Web).

Suggested visualizations

• Top row: KPI cards with sparkline trends (WAU-SI, activation %, avg saves)
• Main chart: Stacked area — weekly WAU-SI by cohort (signup week) to show cumulative adoption and retention
• Secondary: Funnel (view → save → revisit) and bar charts for platform/OS, device, and feature entry point (share, product page)
• Table: recent save failures with error types and counts

Alerts & thresholds

• Save failure rate > 2% (critical) — paging to SRE/dev
• Activation drop > 15% week-over-week — notify PM & designer
• WAU-SI growth < 5% over two weeks — inform growth/product
  Set severity: critical (immediate), warning (48h), info (weekly)

Owners

• PM: product KPIs & thresholds
• Analytics: data accuracy and instrumentation
• Design (you): visualizations, UX interpretation, user-impact analysis
• Eng/SRE: reliability/alerts

Key drill-downs for investigation

• By cohort (signup week) to separate adoption vs retention
• By platform, app version, geo, entry point, user segment (new vs returning)
• Error logs for save failures and timing analysis for time-to-first-save

Design notes: prioritize hierarchy, accessible color palette, compact cards for quick scan, interactive filters (date range, cohort, platform), and exportable reports for stakeholders.

Situation & my role
I was the lead UI Designer on a 10-week redesign of our web app’s onboarding dashboard (Q1, 2023). Goal: increase first-week task completion from 42% to 60%. I led visual design, prototypes, and handoff.

Hypothesis
Cleaner visual hierarchy + a prominent “Start Setup” card and animated affordances would reduce cognitive load and raise completion by 18 points.

Instrumentation & measurement

• Events: impression, click “Start Setup”, task completed (analytics).
• Metrics tracked: first-week task completion, click-through-rate (CTR) of CTA, time-to-first-complete.
• A/B test: 50/50 control vs new UI for 6 weeks, N ~ 12k new users.

Outcome

• CTA CTR rose from 18% → 28% (+10pp) but first-week completion stayed 42% (no lift). Time-to-first-complete increased 10% (from 6.2m → 6.8m).

Analyses run

• Funnel analysis: drop-off concentrated on step 3 (data entry).
• Session recordings & heatmaps: users clicked CTA but hesitated on form fields; mobile keyboard triggered layout shift.
• Segmentation: desktop had small completion lift (+3pp), mobile decreased (-4pp).

Changes attempted

• Iteration 1 (2 weeks): simplified form, inline field validation — ran another 2-week test: mobile drop persisted.
• Iteration 2: deferred non-essential fields to later, added progressive disclosure — small improvement on mobile (+2pp), still below target.

Learnings

• Visual prominence alone didn’t fix UX friction in the form flow. Root cause was interaction on mobile (layout shift, form length). Future priorities: device-specific interaction tests earlier, prototype using real-device keyboard behavior, and include frontend engineers during prototyping to catch layout shifts. I documented assets and recommended analytics additions for step-level timing for future experiments.

Approach summary (as a UI Designer):
I’d treat this as an attribution/confounding problem: run quantitative analyses to segment traffic and temporal patterns, and complement with qualitative UX signals (heatmaps/recordings) to see whether the redesign changed behavior independently of email.

Analyses to run

• Traffic-source & UTM breakdown
  • Compare conversion lift by source (organic, paid, direct, referral, email). If uplift is concentrated in “email” or channels with email UTMs, likely email-driven.
• Click vs non-click cohorts
  • Get the marketing send/click list (or email click events). Compare conversion rates of users who clicked the email vs those who didn’t during the same window.
• Time-series and cohort analysis
  • Plot daily conversion rate before/after redesign and before/after send; check whether uplift begins immediately after send or aligns with release rollout.
• Funnel & page-level analysis
  • Compare drop-off rates on redesigned pages vs previous versions; see where improvements occurred (e.g., CTA clicks, form completion).
• Regression / difference-in-differences
  • Model conversions with binary variables for redesign exposure and email exposure plus controls (device, geography) to estimate independent effects.
• Qualitative signals
  • Heatmaps, session recordings, and feedback to see if UI changes improved clarity or interaction patterns.

When tagging/UTM data is incomplete

• Use click timestamps and landing-page query strings to match sessions to email clicks; reconstruct via server logs or marketing platform click logs.
• Identify users by cookie or user-id across systems (CRM, analytics); stitch events to mark exposure.
• Create proxy cohorts by time-window: users who visited in the hour(s) after the send vs same hour baseline on prior days.
• Retroactively add event tags on backend (e.g., mark user profiles who clicked) so future analysis is cleaner.

Next steps / recommendations

• If unclear, run a controlled experiment or holdout: restrict email to a randomized subset next send while rolling design to everyone, or A/B test UI against control.
• Improve tagging standards and add feature flags to record UI-version exposure.
• Share findings with marketing and analytics, propose instrumenting key conversion events so future attribution is decisive.

This combination quantifies attribution and uses UX evidence to decide whether the visual/design changes actually caused behavioral improvements.