Research and Product Analytics Tools Questions

Proficiency with user experience research and product analytics platforms used to generate qualitative and quantitative insights. This includes hands on use of remote and moderated research tools such as UserTesting Maze and Optimal Workshop and prototyping or usability tools such as Figma; and analytics platforms such as Google Analytics Amplitude and Mixpanel for event tracking funnel analysis cohort analysis retention analysis and experiment evaluation. Candidates should be able to design studies recruit participants select metrics instrument events interpret mixed methods results create dashboards and translate findings into product improvements. Include knowledge of experiment design A B testing basics data quality and how to combine qualitative observations with quantitative signals.

MediumTechnical

21 practiced

Outline a QA plan to validate an analytics implementation that uses Google Tag Manager (GTM) to fire events to Google Analytics and Amplitude. Include manual tests, automated smoke tests, staging checks, expected payload validation, event deduplication checks, and criteria you would use to approve the implementation for production.

Sample Answer

**Overview — goal**Ensure events fired via GTM to Google Analytics (GA4) and Amplitude accurately reflect user behavior so research metrics and segmentation are reliable.

**Staging checks**- Verify GTM container version deployed to staging; enable Preview/Debug.- Confirm GA4/Amplitude keys point to staging projects to avoid polluting prod data.- Walk core flows (signup, search, conversion) in Preview and inspect the GTM console for triggers and tags.

**Manual tests**- Document test checklist: page load, click, form submit, modal open, funnel steps, identity linking.- Use browser devtools + network tab and Amplitude/GMP DebugView to confirm event names, timestamps, and user_id or device_id.- Check UX scenarios: logged-in vs guest, slow networks, mobile viewport.

**Expected payload validation**- For each event, assert required fields present (event_name, user_id/session_id, timestamp, properties). Example properties: product_id, variant, page_section.- Validate data types and value domains (e.g., price numeric, category in allowed list).

**Event deduplication checks**- Trigger same action once and repeat rapidly; ensure single event per intended action.- Simulate page reload/back navigation and verify idempotency (use client_event_id or event_id fields).- Confirm server-side and client-side dedupe strategies align.

**Automated smoke tests**- Implement scripted end-to-end test (Cypress/Playwright) that executes main flows, captures network requests, asserts event names and key payload fields.- Run smoke on each GTM publish pipeline and staging deploy.

**Approval criteria for prod**- 100% of critical events fire in staging with correct payloads.- No duplicated events in repeated-action tests.- Automated smoke tests pass in CI for tagged release.- Stakeholders (research, product) sign off on event definitions and mapping doc.- Monitoring/alerting in place for dropped or malformed events post-deploy.

**Post-release**- Verify first-hour data in GA4 DebugView and Amplitude, compare counts vs staging baseline; escalate if anomalies.

EasyTechnical

28 practiced

Describe best practices for preparing a Figma prototype for usability testing (both remote and moderated). Cover file organization (pages/frames), interaction fidelity choices, linking flows, annotations for moderators/testers, and how to export or share the prototype with testing platforms like UserTesting, Maze, or Lookback.

Sample Answer

**Brief framing**As a Design Researcher I prepare Figma prototypes so usability sessions run smoothly and gather clean behavioral data. Below are practical best practices for file structure, fidelity, linking, annotations, and sharing with platforms like UserTesting, Maze, and Lookback.

**File & page organization**- One page per study or task group (e.g., “Study_A — Onboarding”, “Study_A — Settings”).- On each page: a “1‑Start” frame, then numbered flows (01, 02…) to set clear script order.- Keep master components and style guide on a separate page (“Design System / Tokens”).- Use prefixes in frame names: “TASK_01 — Search”, “CTRL — Empty state”, “DEV — Notes”.

**Interaction fidelity choices**- Match fidelity to research goals: low/intermediate (clickable wireframes) for early discovery; high fidelity for task realism and visual validation.- Use micro‑interactions sparingly—only if they affect task outcome.- Simulate state changes with separate frames rather than complex auto‑animations when you need reliability during recording.

**Linking flows**- Create a single entry frame and explicit exit points to keep sessions on task.- Use Prototype links for primary flows; use hidden hotspots for moderator-only shortcuts (off to the side, marked “MOD_SKIP”).- Ensure “Back” and “Home” controls are present for moderated sessions.

**Annotations for moderators/testers**- Add an “_Instructions” frame at the start with task text, success criteria, and think‑aloud prompts.- In frames, add small sticky notes or use Figma comments prefixed “MOD:” or “NOTE:” for facilitator cues, allowed hints, and expected time per task.- Export a short facilitator script (PDF) mapping tasks to frame IDs.

**Exporting & sharing**- For UserTesting and Lookback: use Figma’s Share → “Prototype” link; set link permissions to Anyone with link (can view). For moderated sessions, copy the URL into Lookback’s interview setup so you can control navigation while recording.- For Maze: export flows to Maze via the Figma + Maze integration (publish selected frames as a Maze test) or export screens as PNGs when you need static tasks.- For high‑fidelity unmoderated tests, ensure device preview (phone/tablet) and disable hotkeys that reveal dev overlays.- Always test the prototype link in an incognito window and run a 5‑minute pilot recording to confirm navigation and recordings.

**Final checklist**- Clear start frame and task instructions- Proper naming and separate design system- Reliable links; moderator shortcuts available- Permissions set; pilot tested on target platform

These practices reduce friction during sessions and improve data quality for actionable insights.

EasyTechnical

37 practiced

You need to run an unmoderated usability test for a web checkout prototype using Maze with a Figma file. Describe the end-to-end process: how you would prepare the Figma prototype and link it to Maze, structure tasks and success criteria, write instructions and screener questions, choose sample size, set Maze settings (time limits, device types), and analyze & report results to stakeholders.

Sample Answer

**Overview (goal)** Run an unmoderated Maze study that validates the web checkout flow in Figma, measuring task completion, friction points, and perceived ease.

**Prepare Figma prototype** - Finalize flows: include start state, primary checkout path, edge states (empty cart, address error, payment failure). - Make hotspots clear and predictable; use Figma prototypes with interactive components and overlays for modals. - Create a “happy path” and 2 common variants (guest vs signed-in). - Use consistent naming and a single entry frame per task. - Publish the prototype and install Maze’s Figma plugin to import frames (or export as prototype link).

**Link Figma to Maze** - From Maze: Create a new test → Import prototype → select Figma file/frames → verify hotspots and navigation in Maze preview. - Add screenshots/alternate flows as follow-up screens if needed.

**Tasks & success criteria** - Write 4–6 tasks phrased as goal-oriented scenarios (e.g., “Buy the blue running shoes, use guest checkout, and complete payment.”) - Success metrics per task: completion rate (primary), time on task, number of clicks, error paths taken, and task-specific checkpoints (e.g., correct shipping option chosen). - Define pass thresholds (e.g., ≥85% completion or <3 min average).

**Instructions & screener** - Study instructions: short context, approximate duration, confidentiality, and consent checkbox. - Task instructions: single-sentence goals; avoid UI hints. Add “you may use the prototype like you would a real site.” - Post-task questions: ease rating (1–7), open-text: “What confused you?” - Screener examples: purchases online in last 3 months (yes), device used (desktop/mobile), exclude internal staff, demographic targeting as needed.

**Sample size & recruitment** - Aim 20–25 participants per user segment (desktop vs mobile or novice vs experienced). For quick unmoderated validation, 15–20 can surface major issues; for comparisons or subgroups, 25+ each. - Recruit via user panel, Prolific, or customer lists; offer appropriate incentives.

**Maze settings** - Device targeting: allow only Desktop or Mobile per test or tag segments. - Time limits: 3–5 minutes per task max; overall session limit ~20–30 minutes. - Randomization: randomize non-critical task order to reduce learning. - Enable screen recordings or session replay if available and get explicit consent. - Require completion of a screener before entry.

**Analyze & report** - Quantitative: task completion, time on task, clicks, drop-off funnel, heatmaps, misclicks. - Qualitative: thematic coding of open responses, notable quotes, screenshots of failure paths. - Synthesize: top 3 usability issues with severity, frequency, and recommended fixes (low/medium/high effort + impact). - Deliverables: 1-page executive summary, slide deck with visuals (funnels, heatmaps, quotes), and a prioritized backlog of design changes. - Present to stakeholders in a 30–45 min readout and run a short prioritization workshop to convert insights into experiments.

Key practice: keep tasks realistic, measure both behavior and perception, and tie recommendations to measurable success criteria for follow-up validation.

MediumSystem Design

30 practiced

Draft an event taxonomy for a product search experience to be implemented in Mixpanel or Amplitude. Provide explicit sample event names (e.g., search_initiated, search_results_shown, result_clicked), required properties for each event (including type and example values), recommended user properties, naming conventions and a versioning strategy that will support funnel analysis, search relevance evaluation, and conversion attribution.

HardSystem Design

23 practiced

Describe an architecture for event-based identity resolution and cross-device tracking that respects GDPR and CCPA. Include strategies for pseudonymization (hashed identifiers), consent flags in event payloads, data retention and deletion flows, and configuration of analytics tools to honor consent while preserving useful aggregated product metrics.

Sample Answer

**Clarify goals & constraints**As a Design Researcher I’d prioritize user trust and lawful analytics: enable cross-device behavioral understanding while minimizing re-identification risk, honoring GDPR/CCPA consent (and opt-outs), and preserving aggregated product metrics for research.

**High-level architecture**- Client SDKs (web/mobile) capture events with local consent check.- Consent & Identity Gateway: evaluates consent flags and maps device identifiers to a pseudonymous internal ID.- Identity Graph (hashed keys + ephemeral linking tokens) in a privacy-layered store.- Event Warehouse with two zones: Raw-pseudonymized and Aggregated-analytics.- Deletion/Retention Service and Consent Audit Logs.

**Pseudonymization & linking**- Hash raw IDs with HMAC-SHA256 and per-tenant rotating salt: - client_id_hash = HMAC_SHA256(salt, client_id)- Store only hashes + short-lived linkage tokens (one-time linking codes sent when user authenticates on a new device). No plaintext IDs in downstream analytics.- Keep minimal metadata (locale, device type) for segmentation.

**Consent flags & payloads**- Each event includes consent flags: marketing:true/false, analytics:true/false, research:true/false.- SDK enforces consent: if analytics==false, block sending; if research==limited, send only aggregated summaries locally.- Gateway enforces and logs consent decisions for audit.

**Retention & deletion flows**- Policy store with TTL per data type; automated retention sweeper removes raw-pseudonymized data after minimal research window (e.g., 90 days), aggregated metrics retained longer.- Deletion request: map user-supplied identifier → hashed keys → remove all linked raw records, revoke linkage tokens, add to suppression list to prevent future re-linking.- Provide attestations and audit trail for compliance.

**Analytics configuration to honor consent**- Split pipelines: events with analytics=true → aggregated metrics; analytics=false → suppressed or used only for necessary compliance logs.- Use differential privacy / k-anonymity thresholds on small cohorts; block reports under N to prevent re-identification.- Expose researcher-friendly views: cohort-level funnels, retention curves, device-agnostic user journeys built from pseudonymous IDs, with sampling and noise where needed.

**Trade-offs & researcher practices**- Trade privacy vs. granularity: rotate salts and limit retention reduces cross-time linkage; coordinate with legal on retention windows that still support longitudinal studies.- For qualitative linking, use explicit opt-in (research:true) and provide transparent consent UI; store consented identifiers in a separate opt-in vault enabling deeper cross-device synthesis.

**Why this matters**This design lets design researchers analyze multi-device behavior and product impact while minimizing re-identification risk, honoring user choices, and providing auditable deletion — maintaining trust and legally defensible research outputs.

Unlock Full Question Bank

Get access to hundreds of Research and Product Analytics Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.