Usability Testing and User Research Questions

Focuses on planning, designing, conducting, and analyzing usability studies and broader user research to surface user needs, pain points, and behavior. Candidates should know how to define research goals and questions, design tasks, recruit representative participants, choose between formative and summative approaches and moderated and unmoderated sessions, and select appropriate test formats such as in home, lab, or remote testing. Includes facilitation techniques, observation and note taking, qualitative and quantitative analysis, synthesis of findings into actionable design recommendations, and how to cite user quotes or data to support product decisions.

HardTechnical

103 practiced

You must design and justify sample size and power analysis for a quantitative usability comparison of two designs. Baseline task completion is 70%; you want 80% power to detect a 10 percentage-point absolute improvement (i.e., 70% -> 80%) with alpha = 0.05. Describe steps, formulas, assumptions, and calculate the approximate sample size per group.

Sample Answer

**Approach (high level)** I’d treat this as a two-sample test of proportions (comparing task completion rates). Steps: define hypotheses, choose one- vs two-sided test, pick alpha and power, use standard sample-size formula for proportions, then adjust for practical factors (dropout, clustering, multiple tasks).

**Hypotheses & assumptions**- Baseline p1 = 0.70, target p2 = 0.80 (absolute difference Δ = 0.10) - Two-sided alpha = 0.05, power = 0.80 (β = 0.20) - Independent users per group, equal allocation, single task per user, no clustering.

**Formula (two-sample proportions)** Use the normal-approximation formula:

text

n per group = [ (Z_{1-α/2} * sqrt(2 * p̄ * (1 - p̄)) + Z_{1-β} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ] / (p2 - p1)^2

where p̄ = (p1 + p2) / 2.

**Plug-in numbers**- p1 = 0.70, p2 = 0.80 → p̄ = 0.75 - Z_{1-α/2} = 1.96, Z_{1-β} = 0.84 Compute pieces:- sqrt(2 * 0.75 * 0.25) = 0.6124 - sqrt(0.70*0.30 + 0.80*0.20) = sqrt(0.21 + 0.16) = 0.6083 Numerator = (1.96*0.6124 + 0.84*0.6083) = 1.71; squared = 2.924 Delta^2 = 0.10^2 = 0.01 n ≈ 2.924 / 0.01 = 292.4 → ~293 users per group

**Notes & practical adjustments**- If you justify a one-sided test (only care about improvement), Z_{1-α}=1.645 reduces n to ~231 per group. - Inflate for attrition/invalid sessions (e.g., +10–20% → target ~322–352 per group). - If measurements are clustered (same participant tries multiple designs) use paired test which has higher power and reduces sample size; compute using paired proportions formula. - If multiple tasks or multiple comparisons, apply corrections (Bonferroni) which increase required n.

Recommendation: plan for ~300 users per arm (round up to ~350 to be safe given real-world loss), or use a paired within-subject test if feasible to reduce sample needs.

MediumTechnical

63 practiced

You need a consent script to record participants during usability testing. Draft the key elements that must appear in the consent script (purpose, recordings, storage, access, anonymization, opt-out) and explain why each is necessary and what practical wording you would use to be clear and compliant.

MediumTechnical

52 practiced

Analytics show a sharp drop-off on step 3 of your signup funnel. Propose a mixed-methods research plan combining quantitative segmentation, event analysis, session replay, and targeted qualitative usability sessions to discover root causes and test solutions. Outline sequencing, samples, and how findings will inform experiments.

Sample Answer

**Goal:** Diagnose root causes of a sharp drop at Step 3 of signup and generate validated design fixes to A/B test.

**1) Clarify success metrics & constraints**- Primary metric: completion rate from Step 3 → finish (funnel conversion)- Secondary: time on step, error events, abandon reason surveys, NPS- Timeline: 4–6 weeks; privacy/legal check for session replay

**2) Sequencing (quant → qual → ideate → validate)**- Week 1: Quantitative segmentation & event analysis- Week 2: Session replay triage + micro-surveys in-step- Week 3–4: Targeted moderated usability sessions + rapid prototypes- Week 5–6: Run A/B experiments informed by findings

**3) Quantitative segmentation & event analysis**- Segment by new vs returning users, device, browser, geo, acquisition source, referral flow- Instrument detailed events: render time, input validation errors, clicks, form abandons- Sample: all users for last 30 days; look for cohorts with statistically significant drop (chi-square / funnel z-test)- Deliverable: prioritized hypotheses (e.g., client-side validation bug on Android Chrome; high load time >3s)

**4) Session replay & micro-surveys**- Replay top 200 drop sessions filtered by segments above; prioritize high-value cohorts- Add 1-question micro-survey on Step 3 (“Why did you leave?” with options + open text) for a ~10–20% response rate- Look for patterns: stuck fields, unexpected fees, payment failures, confusing copy

**5) Targeted qualitative usability**- Recruit 8–12 participants matching problem cohorts (e.g., Android mobile signup, new users from paid channel)- Tasks: complete signup; think-aloud; probe on pain points discovered in replays- Record success rates, time-on-task, observed errors; capture verbatim language for copy solutions

**6) Synthesis → Hypotheses & Design**- Combine quantitative prevalence with qualitative depth to craft 2–4 testable hypotheses (e.g., "Reducing required fields + inline validation will increase completion by 12%")- Prioritize by impact/effort/risk

**7) Experiments**- Implement A/B with clear metrics: conversion lift, error rate, time on step, retention after signup- Run until statistical significance or pre-set test duration; monitor segmentation to ensure fixes help target cohorts

**8) Close the loop**- Post-test analysis: if winning, roll out + monitor long-term metrics; if not, iterate with additional qualitative follow-up.

This plan connects large-scale signal detection to contextual understanding and produces focused, measurable experiments that respect engineering constraints and user privacy.

EasyTechnical

62 practiced

Compare moderated-remote and unmoderated usability testing by listing the main strengths and weaknesses of each. Then give two concrete product scenarios (one favoring moderated-remote, one favoring unmoderated) and explain your rationale based on cost, depth, and scalability.

MediumTechnical

55 practiced

Describe a practical step-by-step approach to analyze open-ended feedback from 200 usability-test participants: from initial sampling and open coding, to iterative theme development, inter-rater reliability checks, quantifying theme prevalence, and presenting representative quotes and evidence to stakeholders.

Sample Answer

**Overview — goal:** Turn 200 open-ended responses into validated, stakeholder-ready insights linking user pain points to design recommendations.

**Step 1 — sampling & prep**- Clean responses, remove duplicates/openers.- Randomly select a 20% stratified sample (by task/segment) for initial coding to capture variation and avoid bias.

**Step 2 — open coding**- I perform line-by-line open coding in a tool (Dovetail/Atlas.ti/Excel) and create short, descriptive codes (e.g., "confusing CTA", "slow onboarding").- Code 2–3 responses at a time, note memos about context and questions.

**Step 3 — build a codebook & iterative theme development**- Consolidate similar codes into candidate themes, define each theme, inclusion/exclusion criteria, and example quotes.- Apply themes to another 20% sample, refine definitions until consistent application.

**Step 4 — inter-rater reliability**- Have a second coder double-code a 20–30% random sample.- Calculate Cohen’s kappa (target > 0.7). Discuss and resolve disagreements, update codebook, re-run on a small adjudication sample.

**Step 5 — full-coding & quantification**- Apply final codebook to all 200 responses (batch in tool or via spreadsheet).- Quantify prevalence: count responses and percentage mentioning each theme; track co-occurrence and segment differences.

**Step 6 — select representative quotes & evidence**- For each theme, pick 2–3 paraphrase-safe quotes: one succinct, one illustrative of root cause, one from a key user segment.- Include screenshots or task timestamp references where applicable.

**Step 7 — present to stakeholders**- Deliver a one-page insight per theme: definition, prevalence (% and n), top pain points, representative quote(s), screenshots, and a concrete design recommendation with impact/effort.- Use visuals: bar chart of theme prevalence, heatmap of co-occurrence, and user journey snippets showing where issues occur.- End with prioritized next steps and A/B or prototype tests to validate design changes.

**Tools & timeframe**- Tools: Dovetail/Atlas.ti, Google Sheets, Figma for storyboards.- Timeline: ~2–3 weeks (sampling + coding + reliability + reporting).

This approach ensures rigor (reliability + quantification), traceability (quotes + artifacts), and actionable outcomes designers and PMs can act on.

Unlock Full Question Bank

Get access to hundreds of Usability Testing and User Research interview questions and detailed answers.

Join thousands of developers preparing for their dream job.