Usability Evaluation: Principles, Heuristics, and Testing Questions

Usability Test Result: Lightweight Template (one row per observed issue / per task)

Participant

• ID: P01
• Profile: Age, role, tech comfort (e.g., 34, retail manager, moderate), prior experience with product
• Recruitment note: (e.g., frequent user / first-time user)

Task

• Task ID & short description: (T1) Add a new product to inventory
• Scenario: (one-sentence context)

Outcome

• Success? (Yes / No / Partial)
• Time on task: mm:ss
• Error count: n

Observed Behavior & Quote

• What I observed: (concise, objective)
• Verbatim quote: "..." (if available)

Pain Point

• Description: (what broke or confused)
• Severity: Low / Medium / High (criteria: High = blocks task; Medium = frustrates/needs workaround; Low = minor friction)
• Frequency: (how many participants encountered this)

Impact & Root Cause (hypothesis)

• Impact on user goal and product metric (e.g., prevents checkout; increases support calls)
• Likely cause: (UX copy, missing affordance, technical lag)

Recommended Next Steps

• Short-term: (quick fix / workaround to test)
• Long-term: (design change / A/B test / dev ticket)
• Priority: P0/P1/P2

Tags / Notes

• Device, browser, accessibility flags, follow-up needed

Example entry
Participant: P03 (28, small-business owner, low technical comfort)
Task: T2 (Apply discount code at checkout)
Outcome: Partial success; Time 02:10; Errors 1
Observed: Clicked “Apply” but nothing changed; retried twice
Quote: "I don't know if that worked. Nothing happened."
Pain Point: Discount field lacks confirmation/error state; no validation shown
Severity: High (blocks purchase for low-tech users); Frequency: 3/6 participants
Impact & Root Cause: Likely missing visual success/error feedback; increases abandonment
Recommended Next Steps: Short-term, show toast "Code applied" on success; Long-term, inline validation + disabled apply button until valid format; Priority: P0
Tags: Mobile iOS, follow-up: confirm backend latency with engineering.

Approach (goal: reduce user-facing errors with max business impact and minimal wasted effort)

1. Clarify objectives & constraints

• Objective: reduce session errors that block key tasks / hurt conversions or retention.
• Constraints: engineering capacity, release risk, telemetry available.

2. Triage pipeline (single source of truth)

• Ingest session-replay events and map each micro-interaction to a canonical "interaction ID" (button X, form field Y, hover menu Z).
• Enrich with analytics: frequency (sessions/day), unique users affected, task context (checkout, onboarding), and error type/severity.

3. Prioritization scoring
   Score each micro-interaction on four factors, each rated on a 0-10 scale so the weighted sum lands on a comparable 0-10 score:

• Frequency (30%): how often it's hit, rated 0-10 (10 = a large share of all sessions hit it).
• Task criticality (30%): 10 for checkout/login/onboarding; 2 for low-value peripheral flows.
• User/business impact (20%): 0-10 based on conversion lift, revenue at risk, support tickets.
• Ease of fix (20%): 0-10, where 10 = trivial fix and 1 = a major effort (so an easy fix scores high, the same way effort is inverted elsewhere in prioritization frameworks).
  Score = 0.3Frequency + 0.3Criticality + 0.2Impact + 0.2Ease. Bucket into: Hotfix (score > 8), Planned Sprint (score 5-8), Monitor/Defer (score < 5).

Worked example: the "Apply Filter" button on the checkout page occasionally submits before all filter fields finish loading, returning an empty results error.

• Frequency = 7 (it shows up in roughly 15% of checkout sessions, a high-frequency interaction).
• Task criticality = 10 (it's in the checkout flow).
• Impact = 7 (drives an estimated 1% conversion loss plus a noticeable bump in support tickets).
• Ease of fix = 9 (a one-line front-end guard, deployable the same day).
  Score = 0.3(7) + 0.3(10) + 0.2(7) + 0.2(9) = 2.1 + 3.0 + 1.4 + 1.8 = 8.3, which is above 8, so this goes straight into the Hotfix bucket.

4. Example thresholds

• Hotfix: score > 8, or any interaction causing task aborts for >1% of users in a key flow.
• Planned: score 5-8.
• Defer: score < 5, or extremely costly to fix for low impact.

5. Execution & validation

• Instrument: add event-level telemetry (interaction ID + error code + task outcome) and tie it back to session recordings.
• Fix rollout: phased, starting with a canary release (ship the fix to a small slice of users, e.g. 5%, watch for regressions, then expand to everyone), or an A/B test where feasible.

6. Tracking impact over time

• Define KPIs: interaction error rate (errors / opportunities), task success rate, conversion funnel steps, time-on-task, customer support volume, and a qualitative watchlist of session-replay samples.
• Dashboard: time series with control vs. treatment cohorts, including confidence intervals (a range around the measured improvement that reflects how much the estimate could shift with a different sample, e.g. "errors down 18%, roughly 10-26%" rather than a bare single number).
• Statistical validation: run pre/post or A/B tests; before declaring success, require a minimum detectable effect (the smallest real improvement you actually care about catching, decided ahead of time) and check the result's p-value (the odds you'd see an improvement this large from random chance alone if the fix did nothing; a common bar is under 5%).
• Monitor regressions: set alerts for increases in error rate or drops in task success.
• Postmortem: log lessons, update detection rules, and repeat triage.

This approach balances user impact, business value, and engineering cost while ensuring measurable validation of improvements.

As a UX Designer, I track these five core usability metrics to guide iteration.

1) Task Success Rate

• What it measures: percent of users who complete a task as intended.
• When useful: early tests to validate core flows (signup, checkout).
• Limitation: doesn't show why failures occur or how much effort was required.

2) Time on Task

• What it measures: median/mean time users take to complete a task.
• When useful: comparing alternative designs or detecting inefficiency.
• Limitation: faster isn't always better, a short time could mean abandonment or guessing rather than ease.

3) Error Rate

• What it measures: frequency of user mistakes or system errors per task.
• When useful: identifying confusing UI elements or unclear affordances.
• Limitation: some "errors" are recoverable and don't equal a poor experience on their own.

4) Satisfaction Scores (e.g., SUS, NPS, post-task rating)

• What it measures: subjective perceived ease, satisfaction, or likelihood to recommend. SUS (System Usability Scale) is a 10-question survey producing a 0-100 usability score; NPS (Net Promoter Score) is a 0-10 "how likely are you to recommend this" survey used as a loyalty proxy.
• When useful: captures overall sentiment and acceptance after changes.
• Limitation: influenced by expectations and mood; needs qualitative context.

5) Abandonment Rate

• What it measures: percent of users who start but do not finish a key flow.
• When useful: spotting drop-off points in funnels (cart, onboarding).
• Limitation: doesn't reveal motivation, requires session replay or interviews to interpret.

Use these metrics together (quant + qual) to diagnose problems and prioritize fixes.

Goal: Determine which of three navigation patterns (A: top nav, B: left rail, C: contextual flyout) yields fastest, most accurate task completion and best subjective satisfaction for primary dashboard workflows.

Study design

• Moderated remote prototype sessions (Zoom + a prototype tool like Figma/UXPin) with think-aloud to capture qualitative issues, plus instrumentation to record task time, clicks, and errors.
• Participants: 60 total (20 per pattern) recruited from the target user persona; between-subjects assignment randomized to avoid learning effects. Size the sample in consultation with a data analyst so the study has a reasonable chance of catching a real, meaningful gap between patterns rather than landing on 20 per group just because it is a round number.

Tasks (representative, realistic)

1. "Find the KPIs for last quarter and export them." (navigation + data access)
2. "Locate the alert for the Sales anomaly on June 14 and acknowledge it." (search/filter + action)
3. "Create a new saved view showing region X and metric Y." (configuration)

Hypotheses

• H1: Mean task completion time differs across patterns (primary).
• H2: Error rate (failed attempts, wrong destination) differs across patterns.
• H3: Pattern C will have higher subjective usability (SUS, System Usability Scale: a 10-item post-task survey producing a 0-100 score) than A or B.

Quantitative measures

• Task completion time (seconds) per task.
• Success rate: binary success within 3 attempts or 2 minutes.
• Error count: misclicks, wrong page visits, need for moderator hints.
• Number of clicks and navigation steps.
• Subjective: SUS score plus a preference ranking.

Procedure

• 10-12 minute intro and consent; 30-35 minute moderated session with 3 tasks; post-task SUS and preference questions.
• Log timestamps, clicks, screen recordings.

Analysis plan and a worked example

• Preprocessing: remove timeouts, flag outliers (values more than 3 standard deviations from the mean), but report the analysis with and without them.
• For the continuous measures (time, clicks, SUS), the test I'd actually run and interpret myself is a one-way ANOVA (analysis of variance: it asks "is the gap between these three group averages bigger than you'd expect from ordinary session-to-session noise, or not?"). If ANOVA says yes, a Tukey HSD post-hoc test tells you which specific pair of patterns differs, since ANOVA alone only tells you "something differs," not which one.
  • Worked example: say average task time comes out to Pattern A = 45s, Pattern B = 38s, Pattern C = 41s. Running an ANOVA on these three groups of 20 returns a p-value under 0.05, meaning a gap this size would be unlikely to appear by chance alone if the three patterns were really identical, so you'd conclude the patterns are not all the same. Tukey HSD then shows B is significantly faster than A, while B-vs-C and A-vs-C are not clearly different, pointing you toward Pattern B on speed even though the overall ANOVA result alone would not have told you which pattern was the standout.
• For success rate and errors (yes/no outcomes per attempt), a chi-square test checks whether the success rate itself differs across patterns. This is the other test I'd run myself, since it's the direct proportions equivalent of the ANOVA question.
• Past those two tests, checking the underlying assumptions, choosing the right substitute test when an assumption fails, and reporting a standardized effect size alongside each result are all a data analyst's discipline, not something to compute by hand. Knowing to ask for that rigor matters more than being able to run it yourself.
• Confidence: use a 95% significance threshold, but weigh effect size and the qualitative think-aloud findings alongside the p-value rather than treating "p < 0.05" alone as the decision.

Interpretation and next steps

• Combine the quantitative differences with qualitative notes from think-aloud to explain why a pattern wins (discoverability, cognitive load).
• If differences are small, run a within-subject follow-up with counterbalancing to increase sensitivity.
• Recommend a pattern for engineering with the estimated impact (e.g., faster task time likely reducing support tickets) and validate with an A/B test in production for external validity.

1. Task Success Rate (completion rate for critical tasks)

• Why: Direct measure of whether users can accomplish primary goals (e.g., checkout, onboarding).
• Instrumentation: Add an event for task_start and task_complete with task_id and user_id. Compute the percent of task_complete over task_start per cohort.
• Alerts: Alert if the rolling 7-day rate drops more than 5 percentage points vs. baseline, or falls below an SLA (service level agreement, an internal target you commit to, e.g., 95% success rate).

2. Time-on-Task (median seconds to complete a task)

• Why: Reveals efficiency and friction; long times often indicate usability issues or confusion.
• Instrumentation: Log timestamps for task_start/task_complete and calculate the median and P90 (the value 90% of sessions fall under, useful for catching slow outliers that a median hides) by task_id, platform, and persona.
• Alerts: Alert on the median increasing more than 30%, or on P90 spiking, to catch regressions.

3. Error Rate (user-facing errors during a task)

• Why: Captures failures that block progress (form validation, API failures).
• Instrumentation: Instrument client-side and server-side error events with error_code, task_id, and severity. Correlate with task flows.
• Alerts: A high-severity error rate above a threshold (e.g., more than 1% of sessions), or a sudden delta, triggers a page.

4. System Usability Scale (SUS) or a short NPS-like UX rating

• Why: Quantifies perceived usability; complements behavioral metrics with subjective sentiment. NPS (Net Promoter Score) is a 0-10 "how likely are you to recommend this" survey used as a loyalty proxy; a short NPS-like question asked specifically about one feature works the same way at a smaller scale.
• Instrumentation: Trigger micro-surveys after task completion or on random sessions; store the SUS score per user and segment.
• Alerts: Average SUS drops more than 5 points over a rolling 14 days, or there's a significant negative shift in comments.

5. Drop-off / Funnel Conversion at Key Steps

• Why: Pinpoints where users abandon multi-step flows (signup, checkout).
• Instrumentation: Track ordered funnel events (step_1, step_2, ...) with session_id; compute conversion between steps and time between steps.
• Alerts: Conversion between adjacent steps drops more than 10%, or absolute conversion falls below target.

Implementation notes:

• Tag events with product_version, device, country, and user_segment to triage regressions.
• Build dashboards showing baseline, trend, and segmentation; create automatic anomaly detection (seasonality-aware) and set on-call escalation rules.
• Pair alerts with automated smoke tests that simulate key tasks to confirm frontend/backend stability before paging.