Usability Evaluation: Principles, Heuristics, and Testing Questions

Brief framing
As a design researcher moderating a 45‑minute usability test I use concise, tested facilitation techniques to build rapport, avoid leading, and surface honest reactions.

Five techniques

1. Warm, neutral intro — 2 minutes

   • Quick personal intro, explain purpose, confidentiality, and that “honest criticism helps us.” Use open, nonjudgmental language to reduce social desirability.

2. Structured think‑aloud prompts

   • Ask participants to vocalize thoughts using neutral prompts (“What are you thinking now?”) and avoid praising specific actions to prevent leading.

3. Task framing with success metrics

   • Give clear tasks and observable goals (e.g., “Find how to reset password — tell me what you expect to happen”) so feedback is about the product, not the interviewer.

4. Handling silent participants

   • Use gentle, low-pressure probes and alternate mediums: “I noticed you paused — could you walk me through what you were thinking?” or give a quick micro-task (click this, describe why) to restart engagement.

5. Managing dominators

   • Use timeboxing and turn-taking: “We’ll spend two minutes on this task; I want to hear a quick reaction from everyone.” If needed, redirect with a neutral facilitator intervention: “Thanks — I want to hear your brief reaction next,” then call on quieter participants.

Why these work

• Builds trust, keeps questions neutral, focuses feedback on behaviour, and balances participation while preserving session flow and validity.

Situation & goal
I want fast, directional usability feedback on a mid-fidelity prototype to validate core flows before a larger study.

Participant selection

• Recruit 6–8 colleagues from non-design teams who resemble target users (e.g., sales reps for a sales tool) — prioritize domain familiarity over interface expertise.
• Screen with 1–2 quick questions (role, frequency of similar tasks).

Consent

• Verbally state purpose, voluntary nature, and that session is ~10 minutes; get verbal consent and permission to take notes/screenshots. Offer opt-out at any time.

Tasks & length

• 3 focused tasks, each 2–3 minutes (total session 8–12 minutes). Use realistic task statements (“Create a quote for customer X and share it”).

Observation method

• Moderator asks participant to think aloud lightly; one observer notes behaviors, pain points, and quotes. Use stopwatch and capture 1–2 screenshots. Avoid coaching.

Deliverables

• 1-page synthesis: key usability issues (severity + evidence), 3 quick recommendations, 5 verbatim quotes, and prioritized fixes for next sprint. Include short video clips for critical failures.

Limitations

• Not representative; biased by colleague context and small N. Good for spotting obvious blockers and language issues, but avoid making definitive product decisions—use it to prioritize follow-up studies or fast fixes, not final launches.

Approach overview
Use a two-sample test for proportions (two-proportion z-test or equivalently chi-square test of independence) to assess whether the increase from 65% (n=200) to 72% (n=200) is unlikely under the null of no true difference.

Test choice & assumptions

• Test: two-proportion z-test (or chi-square).
• Assumptions: independent samples, binary outcome, sufficient sample size so np and n(1−p) ≥ 5 for each group, random or representative sampling. If assumptions fail, use Fisher’s exact test.

Effect-size concept

• Report absolute difference (risk difference): 72% − 65% = 7 percentage points.
• Also report relative effect (risk ratio = 72/65 ≈ 1.108) and standardized effect (Cohen’s h) for comparability across studies.

Power & uncertainty

• Perform post-hoc or prospective power/sample-size calculation to show detectable effect at α=0.05 and power=80% (to assess whether n=200 was adequate).

Reporting & limitations

• Report test used, two-sided p-value, 95% confidence interval for the difference (e.g., 7 pp with CI), and effect sizes (absolute & relative).
• State practical significance in context (is +7 pp meaningful for product goals).
• Limitations: non-random assignment, confounders, multiple comparisons, seasonal effects; if non-experimental, avoid causal claims and suggest controlled A/B or regression adjustment.

Recommendation framework (criteria to decide)

• Goal alignment: Which method best answers the primary question this sprint must resolve (usability, behavioral patterns, or alignment with business goals).
• Risk & unknowns: Prioritize method that reduces the riskiest assumption first.
• Evidence fit: Choose qualitative vs quantitative depending on whether we need depth (why) or breadth (what/where).
• Time & logistics: Two weeks and $5k limits recruitment, facilitation, and analysis time.
• Stakeholder buy-in: Which activity will produce artifacts that influence roadmap decisions quickly.

Decision logic & recommendation

• If the sprint’s question is “Can users complete this new flow?” → run (A) remote moderated usability tests (5 participants). Rationale: direct observation uncovers usability issues, yields concrete design recommendations and recordings for stakeholders. 5 tests fit two-week cadence and ~$3–4k recruitment+moderation, leaving budget for analysis.
• If the unknown is “Where are users dropping off at scale?” → run (B) analytics deep-dive. Rationale: identifies quantitative hotspots and hypotheses to validate; lower cost but won’t reveal why.
• If the problem is organizational alignment or prioritization between initiatives → run (C) stakeholder interviews. Rationale: uncovers constraints, success metrics, and political risks; less direct user insight.

Trade-offs

• Usability tests: high insight quality for interaction issues, moderate cost/time, small sample limits generalizability.
• Analytics: broad coverage, low cost, fast; lacks user intent and contextual richness.
• Stakeholder interviews: helps roadmap alignment and unblock work; risks biasing product toward business needs without user validation.

I’d pick A when shipping-critical UX risk exists, B for pattern detection at scale, and C when we need decision authority or clarify KPIs.

Clarify goals & constraints

• Objective: explain why users keep, churn, or re-engage over 6 months; inform product interventions.
• Success metrics: 6-mo retention, feature adoption curves, qualitative drivers.

Cohort definitions

• Acquisition cohort by week (W1–W24).
• Behavioral cohorts (first 30 days): Active Starter (≥3 sessions/week), Occasional (1–2/week), Trialled-only (used free feature, no conversion), Converted Paid, Early Churn (stop within 30 days).
• Risk cohorts: Declining Engagement (30% drop rolling 14d).

In-app analytics events

• Account created, onboarding completed, key feature X used (with metadata), session start/end, conversion to paid, cancel request, reactivation, feature-specific success event (e.g., completed workflow), NPS prompt shown/responded.
• Capture timestamps, user_id, cohort tags, experimental flags.

Longitudinal qualitative cadence

• Sample 40 participants stratified across behavioral cohorts; schedule interviews at Week 2, Week 8, Month 4, Month 6 (more frequent for churn-risk).
• 30–45 min semi-structured interviews focusing on motivations, friction, context of use, perceived value, and change over time.
• Diary prompts or micro-surveys between interviews for critical moments.

Survey timing

• Baseline (signup): demographics, intent, Jobs-to-be-done.
• Week 4: satisfaction, perceived value, friction.
• Month 3 & Month 6: longitudinal satisfaction, feature awareness, willingness to pay, NPS.
• Triggered micro-surveys after key events (cancel, reactivation).

Linking behavioral cohorts to qualitative themes

• Tag qualitative transcripts with cohort and event timeline; use joint displays (matrix of cohort × themes).
• Code for motivations, barriers, mental models; quantify code prevalence per cohort.
• Run targeted comparisons (e.g., why Occasional vs Active Starter convert) and create journey maps per cohort with exemplar quotes and event timelines.
• Use mixed-methods triangulation: validate analytics patterns with thematic drivers, produce recommendations tied to measurable KPIs.