Usability Evaluation: Principles, Heuristics, and Testing Questions

When to use each (UX Designer perspective)

I choose low-fidelity (lo-fi) early to validate concepts and high-fidelity (hi-fi) later to test realistic interactions and visual design.

Low-fidelity, purpose & example questions

• Purpose: Rapidly explore flows, IA (information architecture: how content and features are organized and labeled), and core assumptions.
• Example questions:
  • "Can you find where to complete X task?" (navigation/flow)
  • "Which of these options best matches your mental model?" (content labeling)

High-fidelity, purpose & example questions

• Purpose: Validate micro-interactions, visual hierarchy, and copy in context.
• Example questions:
  • "Does this control's behavior match your expectations?" (interaction affordance)
  • "Is the error message clear enough to recover?" (usability of messaging)

Trade-offs

• Time: Lo-fi means fast iterations; hi-fi means more prep and polishing.
• Participant behavior: Lo-fi encourages thinking-aloud and tolerates critique; hi-fi triggers more realistic use and aesthetic bias.
• Insights: Lo-fi yields structural, conceptual problems; hi-fi uncovers interaction bugs, accessibility, and perceptual issues.

What SUS measures
The System Usability Scale (SUS) is a 10-item standardized questionnaire capturing perceived usability and learnability of a product. As a design researcher I use it to benchmark user sentiment across designs and track improvements over iterations.

How to convert raw scores

• Score each odd item: response minus 1
• Score each even item: 5 minus response
• Sum the 10 adjusted scores to get a raw total (0-40)
• Multiply by 2.5 to convert to 0-100

raw_total = sum(adjusted_item_scores)   # 0..40
sus_score = raw_total * 2.5             # 0..100

Worked example
Say a participant gives these 10 responses (each on a 1-5 scale):
Item 1 (odd) = 4, Item 2 (even) = 2, Item 3 (odd) = 5, Item 4 (even) = 1, Item 5 (odd) = 4, Item 6 (even) = 2, Item 7 (odd) = 5, Item 8 (even) = 1, Item 9 (odd) = 4, Item 10 (even) = 2.

Odd items get response minus 1: 4-1=3, 5-1=4, 4-1=3, 5-1=4, 4-1=3 (sum = 17).
Even items get 5 minus response: 5-2=3, 5-1=4, 5-2=3, 5-1=4, 5-2=3 (sum = 17).
Raw total = 17 + 17 = 34. SUS score = 34 x 2.5 = 85, which lands in the "Excellent" band.

Typical interpretation thresholds

• 80.3 or above: "Excellent" / A grade (high usability)
• 68: "Average" benchmark
• 51-67: "OK / Fair"
• 50 or below: "Poor" (likely usability problems)

(These thresholds come from industry benchmarking studies and Sauro/Lewis guidelines.)

One limitation
SUS gives a reliable, single-number snapshot but lacks diagnostic detail: it doesn't reveal specific task pain points, why users struggle, or which features cause issues. I always pair SUS with qualitative usability testing (think-aloud, task metrics) to surface actionable insights.

Formative testing happens while a design is still changeable and asks "where does this break and why." Summative testing happens once a design is close to final and asks "does this meet the bar we already set." The names describe the moment in the process, not a specific technique, so the same think-aloud session can be run in either mode depending on when and why you run it.

Formative usability testing

• When: early to mid product lifecycle, on sketches, mid-fidelity prototypes, or early releases.
• Primary objective: discover usability problems and feed them straight into the next design iteration.
• Concrete example: moderated think-aloud sessions with 6 to 8 participants on a mid-fidelity checkout prototype, iterating the design between rounds as issues surface.
• Advantage: fast, actionable qualitative feedback that directly changes the next design decision.
• Key limitation: findings aren't statistically generalizable. A small, convenience-recruited sample tells you problems exist, not what fraction of your whole user base would hit them.

Summative usability testing

• When: late-stage lifecycle, at beta, pre-launch, or after a major release.
• Primary objective: measure usability against a pre-agreed benchmark (success rate, time-on-task, System Usability Scale, a standardized 10-question survey that produces a 0-100 usability score) to judge whether the design meets its goal.
• Concrete example: an unmoderated task-based study with 40 to 50 participants measuring completion and time, to check whether checkout clears a pre-set 90 percent success threshold.
• Advantage: objective, comparable metrics that support a go or no-go decision and let you track improvement release over release.
• Key limitation: it tells you that users failed, not usually why. A 68 percent completion rate is a verdict, not a diagnosis, so a low summative score almost always needs a follow-up formative round to explain it.

Worked example of why sample size matters here
Suppose a formative round runs 6 participants and 2 of them fail a task, a 33 percent failure rate. That's a strong signal to fix something, but it's not a statistic. With only 6 people, each participant is about 17 percent of the whole result, so "2 of 6 failed" and "3 of 6 failed" look similar in raw numbers but describe very different severities. A summative test with 50 participants showing 34 failures (68 percent) is a different kind of claim: with that many people, the percentage is stable enough to compare against a fixed threshold.

Trade-offs and pitfalls
The most common mistake is treating a formative test's completion percentage as if it were a summative pass or fail number. A director asking "so what percent failed" after a 6-person formative round is asking a summative question of a formative study; the honest answer is a direction (most people struggled at this one step), not a rate.

Direct answer
Think-aloud is a moderated method where a participant says what they are thinking, expecting, and deciding out loud while actually performing a task, so the researcher gets the reasoning behind an action and not just the action itself. It is most useful early, on prototypes or complex flows where you need to understand why someone got stuck, not just whether they did.

Structured elaboration

• When it is most useful: early-stage prototype testing to catch usability problems and mismatched mental models; exploring how people reason through a genuinely complex flow, like account settings or a multi-step form; and comparing two competing designs to see real-time reactions rather than a stated preference after the fact.
• Benefits: it reveals the reasoning behind an action, not just the outcome; it surfaces the participant's own words for describing a problem, which is often more useful to a design team than a researcher's paraphrase; and it is cheap to set up compared to instrumented or eye-tracking methods.
• Limitations: narrating your thoughts is not a natural way to use a product, so it can change behavior (this is called reactivity), some participants simply struggle to verbalize a running internal process and go quiet exactly when a task gets hard, and it is a poor fit for measuring true, unprompted task time since talking slows people down.
• Encouraging verbalization without leading: give a short neutral demonstration before starting ("please say out loud what you are thinking as you go, there are no right or wrong answers here"); if someone goes quiet, use a fixed, neutral prompt like "what are you thinking right now" rather than a leading one like "did you expect that to happen"; and remind participants explicitly that the design is being tested, not them, since that framing reduces the pressure to perform or apologize.
• Two interpretation pitfalls worth naming directly, because they are easy to miss when reading a transcript after the fact: first, verbalization is not a live readout of cognition, it often lags behind or reconstructs a reason after an action rather than reporting the actual thought that drove it, so treat a participant's stated reason as a clue to investigate, not as ground truth about why they clicked something. Second, silence does not mean approval. Because narrating is itself hard, participants tend to go quiet precisely when a task gets difficult, which is exactly the moment you most need signal; reading a quiet stretch as "nothing wrong here" gets the interpretation backwards.

Worked example
A participant testing a settings screen says, "I'm looking for where to turn off notifications, I think it's under Account," clicks Account, pauses for 8 seconds without speaking, then clicks back and tries Preferences instead. Read literally, the verbalization ends at "I think it's under Account," but the 8-second silent pause is the actual usability signal, that is where they hit friction inside a screen that did not confirm or deny their guess quickly enough. A researcher who only counts what was said out loud would log "found it under Preferences, no issues," missing that Account's lack of feedback is what pushed them to guess again.

Trade-offs and pitfalls
Think-aloud works best paired with a brief retrospective probe after the task ("you paused there, what was going on") to recover reasoning the live narration missed, especially during those silent stretches. It is a poor match for tasks where you specifically need an unprompted, natural time-on-task number, since the act of narrating adds real seconds to every task.

Usability testing means watching real people try realistic tasks on a product so you can find where it actually breaks, rather than guessing from opinions alone. Moderated and unmoderated are two ways to run that observation, and they differ mainly in whether a researcher is present live to ask "why."

Core differences

What each produces

• Moderated: why users struggle, their mental models, and how they try to recover from a mistake.
• Unmoderated: quantitative task success and time, and common drop-off points across a broader sample.

Worked example
Suppose production analytics show 68 percent of mobile checkout sessions complete, versus 91 percent on desktop, a 23 percentage-point gap. A moderated test with even 5 or 6 mobile users watching them attempt checkout will usually surface the reason (a required field hidden below the fold, say) faster than an unmoderated study would, because you can ask "what did you expect to happen there" the moment it happens, instead of inferring it later from a recording.

Two scenarios for each
Choose moderated when: (1) validating a complex, early-stage flow like onboarding or checkout where understanding "why" matters more than a number, or (2) running accessibility-focused sessions that benefit from think-aloud and real-time follow-up.
Choose unmoderated when: (1) validating a small UI change across 100+ users to check whether a completion rate actually moved, or (2) collecting real-world context, different devices, networks, and locations, that a lab or video call can't replicate.

Trade-offs and pitfalls
It's tempting to default to unmoderated because it's cheaper and faster, but if the real unknown is "why," not "how many," unmoderated data leaves you guessing at the cause from indirect clues, a spike in time-on-task, a replay clip, rather than just asking the person directly.