Usability Evaluation: Principles, Heuristics, and Testing Questions

Brief overview
As a UX Designer I use a checklist to run smooth 45-minute remote-moderated usability tests: prepare tech, participants, consent/recording, contingencies, and privacy safeguards.

Tools (recommended)

• Video + recording: Zoom or Microsoft Teams (cloud recording)
• Screen-share & remote control: Lookback, UserTesting, or Zoom
• Prototype: Figma/Adobe XD with share link
• Scheduling & reminders: Calendly + Google Calendar
• Notes & analysis: Otter.ai (transcripts), Airtable/Notion for session notes

Participant setup (pre-session email)

• Confirm date/time, timezone, and duration (45 min)
• Ask to use a laptop and latest browser; provide test link
• Include quick tech-check steps (camera, mic, internet >10 Mbps)
• Ask to close other apps and use headphones
• Share consent form and study purpose with link to sign before session

Consent & recording

• Begin session by restating purpose, voluntary nature, and recording use
• Obtain verbal consent on-record and note timestamp
• Offer option to pause/stop recording anytime
• Clarify how recordings/transcripts will be used and stored

Contingency plans

• If participant loses connection: try reconnecting for 5 minutes, then reschedule remaining time
• If screen-share fails: ask participant to narrate while you guide via shared prototype link or use remote control
• Backup: phone call + step-by-step verbal test
• Have spare participant(s) or buffer slots in schedule

Data privacy

• Limit recording access to core research team; store in encrypted drives (Google Workspace with 2FA or company S3 with encryption)
• Anonymize transcripts; remove PII before sharing
• Retention policy: state duration (e.g., 12 months) and deletion process
• Ensure vendor tools are GDPR-compliant if applicable

Session logistics (timeline for 45 min)

• 0–5 min: welcome, consent, tech check
• 5–10 min: warm-up/background questions
• 10–35 min: tasks & think-aloud (3–4 tasks)
• 35–42 min: debrief & follow-ups
• 42–45 min: wrap-up, compensation instructions

This checklist ensures reliable sessions, clear consent, resilient contingencies, and protected participant data.

Definition & why it matters
Sample bias occurs when your participant mix (here: power users) differs from the product population, so findings won’t generalize. As a UX researcher I both detect bias and correct or qualify results.

Detecting bias

• Compare sample vs product population on key dimensions (frequency, tenure, feature usage) using analytics and user registry.
• Run chi-square or proportion tests to check differences.
• Inspect behavioral metrics (session length, feature depth) to spot overrepresentation.

Correcting / mitigating

• Weighting: re-weight participant responses to match known population proportions.
• Targeted follow-ups: recruit quotas for underrepresented cohorts (casual users) and run focused usability sessions.
• Triangulation: validate qualitative findings with analytics (funnel conversion, error rates), A/B results, and support logs.
• Report uncertainty and sensitivity: show results with/without weights, and confidence intervals.

Weighting example (two cohorts)
Population proportions: casual 70%, power 30%
Sample proportions: casual 30%, power 70%

Weight formula:

w_i = population_prop_i / sample_prop_i

Intuition: boost underrepresented groups, shrink overrepresented ones.

Calculate weights:

w_casual = 0.70 / 0.30 = 2.333
w_power  = 0.30 / 0.70 = 0.4286

Apply to task success rates (casual 85%, power 95%):

weighted_score = (sample_prop_c* w_c * score_c + sample_prop_p * w_p * score_p) /
                 (sample_prop_c * w_c + sample_prop_p * w_p)

Numbers:

• Numerator ≈ 0.32.3330.85 + 0.70.42860.95 ≈ 0.595 + 0.285 = 0.880
• Denominator ≈ 0.32.333 + 0.70.4286 ≈ 1.0
• Weighted success ≈ 88.0%

Practical notes

• Weighting assumes subgroups are internally representative; follow-up sessions are safer when possible.
• Triangulate: if analytics show casual users struggle in the same flows, confidence increases.
• Report methods transparently (how weights calculated, sample sizes, limits).

Approach overview
I’d convert rich session transcripts into quantitative, transparent signals so PMs and engineers can prioritize. Steps: coding → scoring → weighting → uncertainty → visualization.

1) Code transcripts

• Develop a codebook with theme definitions, inclusion/exclusion rules, and example quotes.
• Two coders independently code all 15 sessions; resolve disagreements via discussion and a third reviewer.
• Report inter-rater reliability (Cohen’s kappa) to show coding consistency.

2) Convert themes to counts/scores

• For each theme, record:
  • Frequency: number of distinct participants (out of 15) who experienced the theme.
  • Incidence: total mentions (for repeatability signal).
• Create a normalized frequency: F = participants_with_theme / 15.

3) Weight severity and frequency

• Assign severity per participant-mention on a simple scale (1 = minor annoyance, 2 = moderate disruption, 3 = task-blocker). Coders assign severity when tagging.
• Compute theme score:
  • ThemeScore = (sum of severities across participants) / 15 — yields a 0–3 scale reflecting prevalence + severity.
• Alternative weighted formula:
  • ThemeScore = F * avg_severity (range 0–3).

4) Present uncertainty & confidence

• For binary presence per participant, compute 95% Wilson confidence intervals for F (better than normal approx with small n).
• For ThemeScore, bootstrap participant-level sampling (resample participants with replacement 5,000x) to get 95% CI for the score.
• Show kappa and number of conflicting codes to contextualize reliability.

5) Visuals & prioritization

• Ranked table with: theme, participants, F, avg_severity, ThemeScore, 95% CI, kappa.
• Bar chart or heatmap of ThemeScore with error bars.
• Prioritization quadrant: Impact (ThemeScore) vs Implementation Effort (from engineering estimate) to drive trade-offs.
• Include sample quotes (2 per theme) to preserve qualitative nuance.

Why this is robust

• Combines frequency and severity so low-frequency blockers still surface.
• Uses inter-rater reliability and CIs to communicate confidence.
• Bootstrapping and Wilson intervals are appropriate for small-N studies (15 users).
• Visuals + quotes keep stakeholders grounded in users’ voices while enabling data-driven prioritization.

Overview & goal
I would run an A/B experiment to test whether the new documentation search UI increases developers’ ability to find usable code examples (task completion), while also measuring speed, quality signals, and content implications.

Hypotheses

• Primary (H1): The new UI increases task completion rate for finding a usable code example vs. control.
• Null (H0): No difference in task completion.
• Secondary: The new UI reduces time-to-first-useful-snippet, increases code-copy events, and improves subjective satisfaction.

Primary & secondary metrics

• Primary: Task completion rate per search session — defined as a session where the developer performs a recognized “success” action within 5 minutes (e.g., copies code, opens a full example, clicks “try/run”, or marks “helpful”).
• Secondary:
  • Time-to-success (seconds from first query to success action)
  • Code-copy rate (copies per session)
  • Example open rate (unique example views)
  • Query reformulation rate (lower is better)
  • Self-reported satisfaction (optional micro-survey)
  • Long-term retention: return rate within 7 days

Instrumentation points

• Experiment assignment (bucket id) stored in user profile/session.
• Search query logs: query text, timestamp, result ranking, variant served.
• Interaction events with result items: impression, click, example open, expand, copy, run/try, scroll depth, dwell time.
• Session identifiers, user id (hashed/anonymized), device, locale, account type, referrer.
• Micro-survey responses (if triggered).
• Downstream signals: follow-up navigation to SDK/docs, issue creation, or repo visits.

Log each event with timestamp and bucket id to enable sequence analysis.

Sample size logic

• Estimate baseline completion p0 (e.g., 0.40). Target minimum detectable effect (MDE) = 0.05 (5 percentage points). Using alpha = 0.05 and power = 0.8, approximate N per arm ≈ 2,000–3,000 sessions. (I’d compute exact N with a power calculator during planning using p0, MDE.)
• Run experiment until reaching required N per arm plus at least one full product cycle (e.g., weekdays/weekend) to capture behavioral variance.
• Pre-register analysis plan and stop rules to avoid peeking bias.

Handling bots and noisy logs

• Filter known crawlers by user-agent and IP ranges.
• Require client-side JS heartbeats and interaction events; drop sessions without valid JS events.
• Apply heuristics: minimum session duration (>5s), minimum number of distinct events, realistic interaction patterns (mouse/keyboard), and rate-limiting to exclude scripted traffic.
• Deduplicate rapid repeated events; aggregate by session id to avoid double-counting.
• Run sensitivity analyses: with and without strict bot filters to check robustness.

Analysis plan

• Intention-to-treat (ITT) for primary metric.
• Use bootstrap or binomial test for completion rate; report absolute and relative lift with 95% CI and p-values.
• For time metrics, use median and survival analysis (Kaplan–Meier) to handle censored sessions.
• Segment analyses: new vs. returning users, novice vs. experienced (based on prior doc visits), language/SDK, device.
• Validate instrumentation by sampling event traces and running QA queries to confirm bucket consistency.

Reporting

• Engineers: detailed dashboard with raw and aggregated metrics, event logs, instrumentation checklist, SQL queries, and failure cases. Include rollout plan and performance impact.
• Content authors: condensed insights showing which queries benefit, heatmaps of result clicks, examples with high/low conversion, query-to-example mapping gaps, and suggested content changes (rewrite, add snippet, surface short examples). Provide prioritized list of top 20 queries with low success rates.
• Executive/PM summary: one-page with key metric lifts, CI, segments, confidence, recommended next steps (rollout, iterate, content work).

Example success criteria

• Statistically significant positive lift on primary metric (p < 0.05) and no regression in secondary quality signals (e.g., increased refunds of docs or support tickets). If positive, staged rollout + content fixes; if mixed, iterate on UI or content and re-test.

This plan emphasizes clear success definitions, robust instrumentation, noise filtering, and actionable reporting tailored to engineers and content maintainers.

Task 1 — Create an account with email only
Prompt: "Please sign up for a new account using your email address. Skip any optional steps and finish at the point where you can access the app's main dashboard or landing area."
Success criterion: The participant completes email entry, submits any required fields, confirms account creation, and reaches the app's main dashboard or confirmation screen.
Moderator probes if stuck:

• "What are you expecting to happen after you submit your email?"
• "Do you see any errors or required fields you’re unsure about?"
• "Would you look for an alternative sign-up method or help link?"

Task 2 — Add a profile photo and short bio (optional)
Prompt: "After signing up, add a profile photo and write a two-sentence bio; you may skip if you prefer, but try both adding and skipping paths."
Success criterion: The participant successfully uploads/captures a photo and saves a bio, and also demonstrates the skip flow (or explains why they skipped).
Moderator probes if stuck:

• "What makes you unsure about adding a photo or bio?"
• "Can you show where you'd expect to edit or remove this information later?"
• "Is there anything preventing you from uploading a photo now?"

Task 3 — Verify your email
Prompt: "Verify your account by completing the email verification step—open the verification email and follow the link or enter the code provided."
Success criterion: The participant locates the verification email, completes verification (clicks link or enters code), and returns to a verified account state.
Moderator probes if stuck:

• "Check your inbox and spam—what do you see?"
• "If you didn't receive it, what would you try next?"
• "Does the verification message give you the information you need to proceed?"