Design Impact and Measurement Questions

Covers how design decisions produce measurable product and business outcomes and how to demonstrate that value. Candidates should be able to define success criteria and translate design goals into clear key performance indicators such as conversion rate, task completion and success rates, engagement and session duration, user retention, adoption and feature usage, Net Promoter Score, customer satisfaction, accessibility compliance, and revenue or funnel impact. Topics include establishing baselines, instrumenting analytics and event tracking, designing and running controlled experiments and A B tests, and analyzing both quantitative data and qualitative feedback from usability testing, user interviews, surveys, and heuristic evaluations. Candidates should be able to attribute outcomes to design changes while accounting for confounding factors, perform cohort and segmentation analysis, calculate the return on investment of design changes, set up dashboards and reporting, iterate based on measured outcomes, and communicate impact and trade offs to product and business stakeholders. Interviewers should expect concrete examples of designs that succeeded or failed, how metrics were chosen and measured, how experiments were instrumented and interpreted, and how learnings influenced product strategy.

EasyTechnical

23 practiced

List concrete accessibility compliance metrics you would track to measure UX accessibility across a product. For each metric describe how you'd measure it (automated tools, manual audit, user testing), frequency of measurement, and one limitation of that metric.

Sample Answer

**Overview**Below are concrete accessibility compliance metrics a UX Designer can track, with how to measure them, recommended frequency, and one limitation each.

1) Automated WCAG pass rate (per page/component)- How: Run axe-core, Pa11y, or Lighthouse CI across build/deploy pipelines.- Frequency: Every CI build + weekly full-site sweep.- Limitation: Catches only programmatic issues (contrast, missing alt) — false positives/negatives common; doesn’t measure usability.

2) Manual audit coverage (% of key flows reviewed)- How: Expert heuristic audits against WCAG success criteria for primary user flows (signup, checkout, dashboard).- Frequency: Quarterly or after major releases.- Limitation: Time-consuming and subjective; limited scope vs whole product.

3) Keyboard-only task success rate- How: User testing with participants completing representative tasks using only keyboard; measure completion and time.- Frequency: Every major release or quarterly.- Limitation: Small sample sizes may not generalize.

4) Screen-reader task success & error rate- How: Usability tests with screen-reader users (NVDA, VoiceOver); record task completion, errors, and qualitative feedback.- Frequency: Biannually or for high-impact features.- Limitation: Recruiting specialized participants is harder; tests take longer.

5) Color contrast compliance (% elements passing AA/AAA)- How: Automated scans + spot manual checks for dynamic content.- Frequency: Weekly automated + manual checks during design reviews.- Limitation: Contrast passes don’t guarantee perceptual clarity or information hierarchy.

6) ARIA & semantic markup issues (count)- How: Linting tools (eslint-plugin-jsx-a11y) and code reviews.- Frequency: Per PR + monthly reports.- Limitation: Tool flags may not reflect correct semantic intent or runtime behavior.

7) Accessibility bug backlog and MTTR (mean time to resolve)- How: Track A11y tickets in issue tracker; measure count and average time-to-fix.- Frequency: Weekly dashboard.- Limitation: Depends on triage priority; backlog can hide quality of fixes.

8) Real-user accessibility incidents & NPS for assistive tech users- How: Collect feedback via in-app surveys targeted to assistive tech users and support tickets.- Frequency: Ongoing; monthly summary.- Limitation: Self-selection bias and low response rates.

Use this mix of automated, manual, and user-driven metrics to get complementary visibility: automated for scale, manual for judgment, and user testing for real-world accessibility.

HardTechnical

23 practiced

Estimate the potential business impact (revenue/upside and costs) of making a SaaS product fully WCAG AA compliant. Describe the assumptions you would make, the data you'd seek, and a simple model to produce a conservative and aggressive scenario for execs.

MediumTechnical

23 practiced

Write pseudo-SQL (or describe in SQL) how you would compute weekly retention for signup cohorts (week 0 signup cohort, week 1 retention, week 2 retention, etc.). Describe the tables and fields you assume, and any edge cases you would handle (timezones, duplicate events, partial weeks).

Sample Answer

**Approach (brief)** Compute weekly signup cohorts (by ISO week) and for each cohort week measure percent of users who had an event in subsequent weeks. Use a users table + events table, dedupe events per user-week, align weeks by user’s signup-week.

**Assumed tables / fields** - users(user_id, signup_ts TIMESTAMP) - events(user_id, event_ts TIMESTAMP, event_name)

**SQL (pseudo)**

sql

WITH signup AS (
  SELECT user_id,
         DATE_TRUNC('week', signup_ts AT TIME ZONE 'UTC') AS cohort_week
  FROM users
),
events_week AS (
  SELECT user_id,
         DATE_TRUNC('week', event_ts AT TIME ZONE 'UTC') AS event_week
  FROM events
  GROUP BY user_id, DATE_TRUNC('week', event_ts AT TIME ZONE 'UTC')  -- dedupe multiple events same week
),
cohort_activity AS (
  SELECT s.cohort_week,
         e.event_week,
         COUNT(DISTINCT s.user_id) AS active_users
  FROM signup s
  JOIN events_week e ON s.user_id = e.user_id
  WHERE e.event_week >= s.cohort_week
  GROUP BY 1,2
),
cohort_sizes AS (
  SELECT cohort_week, COUNT(*) AS cohort_size FROM signup GROUP BY 1
)
SELECT c.cohort_week,
       EXTRACT(week FROM e.event_week - c.cohort_week) AS week_number,  -- 0 = signup week
       e.active_users,
       cs.cohort_size,
       ROUND(100.0 * e.active_users / cs.cohort_size,2) AS retention_pct
FROM cohort_activity e
JOIN cohort_sizes cs ON e.cohort_week = cs.cohort_week
ORDER BY c.cohort_week, week_number;

**Edge cases & UX notes** - Timezones: normalize to UTC or product’s business timezone; document choice. - Duplicate events: dedupe per user-week to avoid inflating retention. - Partial weeks: exclude current ongoing week from retention or mark as partial; show both raw counts and completeness flag. - New user activation definitions: consider using meaningful event (e.g., "first_task") instead of any event.

Use these cohorts to inform UX: identify drop-off weeks, prioritize onboarding improvements, and run targeted experiments.

HardTechnical

26 practiced

You deployed a UX redesign to 50% of users and want to measure its long-term retention impact over 12 months. Explain which statistical methods you would use beyond a vanilla A/B test to estimate causal impact over time, accounting for seasonality, cohort effects, and attrition.

HardTechnical

21 practiced

A UI change produced a statistically significant increase in conversions, but backend pricing logic also changed during rollout. Describe a rigorous approach to attribute the uplift correctly using instrumentation logs, server-side changes, and statistical controls. Include concrete analyses and checks.

Sample Answer

**Clarify the question & goal**

I’d state the hypothesis: “UI change caused conversion uplift.” Then list potential confounder: concurrent backend pricing change. Goal: rigorously separate UI effect from pricing.

**Data & instrumentation audit**

- Verify feature-flag logs and rollout timestamps (client and server) for UI and pricing changes; extract authoritative server-side change events.- Join session/user events to server logs by user_id + timestamp to map whether each session saw UI A/B and which pricing logic applied.- Validate instrumentation: check missing events, duplicate IDs, and time drift between front-end and backend.

**Analysis plan (quantitative)**

1. Define cohorts at session-level: (UI treatment vs control) × (pricing-old vs pricing-new). 2. Run balance checks on pre-period behavior and covariates (traffic source, device, geography). Failures indicate confounding.3. Primary estimate: regression with interaction - Outcome: conversion (binary or value) - Predictors: UI_treatment, Pricing_new, UI_treatment × Pricing_new, controls (user fixed effects or demographics), time fixed effects - Coefficient on UI_treatment (adjusted) is UI-only effect; interaction tests non-additivity.4. Robustness: - Difference-in-differences using pre/post windows for groups that later received pricing change. - Instrumental variable if pricing rollout quasi-random (use rollout batch ID as instrument). - Subset analysis: restrict to users who experienced pricing unchanged during their exposure window. - Placebo tests: run same regression on pre-rollout period expecting null effect. - Sensitivity to omitted confounders (e.g., Rosenbaum bounds).

**Checks & diagnostics**

- Pre-trend plots for conversion by cohort.- Power and sample-size re-check given observed variance.- Check heterogeneity by segment (new vs returning users).- Inspect session lengths and drop-off funnels to ensure UI didn’t change engagement patterns in ways mis-measured as conversions.

**Qualitative follow-up**

- Review session recordings and surveys for users with different pricing to detect perceived price-driven behavior vs UI-driven clarity/confidence.- Usability testing on isolated UI with controlled pricing to observe causal mechanisms.

**Decision rule**

If adjusted UI coefficient remains significant and robust across specs and placebo tests, attribute uplift to UI; if pricing explains effect or interaction dominates, conclude pricing responsible and recommend decoupled rollout next time plus improved instrumentation and coordinated experiment windows.

Unlock Full Question Bank

Get access to hundreds of Design Impact and Measurement interview questions and detailed answers.

Join thousands of developers preparing for their dream job.