Research Design, Methodology, and Rigor Questions

Primary outcomes (direct success signals)

1. Weekly Active Users (WAU) — metric: change in unique users with ≥1 session/week

• Maps to: adoption and retention impact of recommendations driving re-engagement
• Caveat: growth could come from marketing or seasonality; attribute with experiment/attribution window

2. Recommendation-driven DAU/WAU ratio — metric: share of active users who engage via a recommendation click

• Maps to: how much the feature changes user behavior vs. organic paths
• Caveat: clicks ≠ value; shallow interactions inflate metric without long-term benefit

3. Time to meaningful action — metric: median time from session start to completing a target (e.g., content consumed, purchase)

• Maps to: whether recommendations shorten discovery friction and increase task completion
• Caveat: shorter time could mean lower exploration; segment by successful vs. aborted flows

Secondary outcomes (quality and business health)

4. Engagement depth on recommended items — metric: completion rate / dwell-time / downstream conversions for recommended content

• Maps to: recommendation relevance and downstream monetization
• Caveat: content type differences skew dwell-time; normalize by category

5. User satisfaction / qualitative sentiment — metric: NPS or in-product rating + thematic feedback from usability sessions

• Maps to: perceived usefulness and trust in recommendations (critical for long term retention)
• Caveat: self-report bias and low response rates; combine with behavioral signals

For each metric I’d define segments (new vs. returning, power users), run A/B tests, and triangulate quantitative signals with qualitative usability interviews to explain “why” behind the numbers.

Convenience sampling
Definition: Recruiting participants who are easy to reach (e.g., colleagues, existing customers, or intercepts).
Use-cases: quick formative usability tests to iterate on wireframes; early discovery interviews to validate assumptions before investing in a larger study.
Limitation: high risk of bias and low generalizability—insights may not reflect broader user segments, so decisions based solely on these samples can mislead prioritization.

Stratified sampling
Definition: Intentionally selecting participants so key subgroups (strata) — e.g., age, device type, or usage frequency — are represented proportionally.
Use-cases: benchmarking UX metrics across novice vs. power users; segmentation research to compare needs of enterprise vs. SMB customers.
Limitation: requires accurate population data and more recruitment effort; mis-specifying strata can still produce unrepresentative results.

Random sampling
Definition: Selecting participants from the target population such that each member has an equal chance of inclusion.
Use-cases: surveys to estimate feature adoption rates with statistical validity; A/B test recruitment to ensure comparable groups.
Limitation: often costly or impractical for qualitative studies and can produce low yield of target behaviors, requiring larger sample sizes to be useful for product decisions.

Overview / Goals
I would establish a staged onboarding: discovery → vendor selection → pilot → scale. Goals: high-quality eye-tracking/biometric signals, participant comfort, data privacy, and reproducible deliverables for research analysis.

Vendor selection criteria

• Domain experience with behavioural research and IRB-friendly protocols
• Accuracy specs: sampling rate, spatial accuracy (e.g., <0.5°), latency
• Supported sensors (eye-tracker model, heart-rate, GSR) and validated algorithms
• Data format/export (raw + processed), APIs, SDKs, platform compatibility
• Security: SOC2/GDPR compliance, data encryption, breach response
• SLAs, support, and references from similar studies

Pilot success metrics

• Data yield: % usable sessions ≥ 90%
• Signal quality: blink/artifact rate < 5%; tracking ratio > 95%
• Participant comfort: NPS ≥ 8 and <10% report discomfort
• Time: setup + calibration ≤ 5 minutes average
• Repeatability: key metrics variance within expected bounds across runs

Data formats & delivery

• Raw (timestamped gaze/x,y, pupil), synced biometrics (HR, GSR), event logs
• Provide CSV/JSON + standardized schema and sample ETL scripts
• Timebase alignment (ms) and sampling-rate metadata
• Secure S3 transfer or API with encrypted at-rest/in-transit

QA procedures

• Automated validation (timestamp continuity, sampling gaps, unit tests)
• Visual QA: heatmaps, scanpaths, overlay on stimuli
• Manual spot checks and cross-check with video recordings (if consented)

Participant experience checks

• Standardized calibration flow, accessible instructions, fallback options
• Brief post-session survey (comfort, distraction, device acceptance)
• Alternate modes for participants wearing glasses/contact lenses

Contractual clauses

• Data ownership: research team retains ownership; vendor limited-use for ops
• Data retention & deletion schedule; right to audit
• Liability & indemnity for data breaches; breach notification timelines
• Performance SLAs and remedies for data quality failures
• IP: models trained on our data require our consent and defined licensing
• Compliance warranties (GDPR/CCPA/IRB) and participant consent support

I’d capture these in a checklist and run a 20–30 participant pilot before full deployment.

Overview & goals
I’d treat this as iterative scale development: create defensible items, verify content validity, test dimensionality, ensure reliability, and decide on IRT if we need item-level precision or adaptive use.

Item generation

• Literature review on task flow, cognitive load, interruptions, friction points.
• Generate 30–40 candidate items covering causes, experience, recovery, and impact; use simple tense and 5- or 7-point Likert stems.
• Map items to a content matrix to ensure coverage.

Expert review & content validity

• Convene 5–7 experts (UX researchers, psychometrician, product manager). Collect CVI ratings for relevance/clarity; revise or drop items with low CVI.

Cognitive interviews

• 8–12 participants across personas; think-aloud probing for interpretation, ambiguity, and response mapping. Revise wording accordingly.

Pilot study

• Deploy revised ~18–24 items to N=250–400 diverse users (rule: ≥10 respondents per item).
• Pre-register analysis plan.

EFA → CFA

• Split sample. Run EFA (oblique rotation) to identify factors, drop cross-loading or low-loading (<.40) items.
• Use CFA on holdout sample to test model fit (CFI ≥ .95, RMSEA ≤ .06, SRMR ≤ .08). Iterate.

Reliability & item reduction

• Target Cronbach’s alpha and McDonald’s omega ≥ .80 per factor. Examine item-total correlations and factor loadings; reduce to 12 items balancing reliability and coverage.

IRT decision

• Apply IRT (graded-response model) if: precision across trait levels matters, computerized adaptive testing expected, or items show local independence. Use IRT to examine item information, DIF, and measurement precision; retain items with high information where sample-relevant.

Outcome

• Deliver final 12-item scale, scoring rules, validation evidence, and recommendations for fielding and future monitoring.

Overview & legal basis

• Treat biometric data as highly sensitive; use explicit informed consent as legal basis and document controller/processor roles. Conduct a DPIA before sharing.

Data minimization

• Share only variables required for the research question (e.g., gaze coordinates + timestamps rather than raw video).
• Remove irrelevant PII (names, email) and trim recordings to essential segments.

Consent language (sample lines)

• “You consent to share anonymized/pseudonymized biometric (eye-tracking, heart-rate) and screen-recording data with University X for research on usability. Participation is voluntary; data will be stored for 2 years and then deleted. You may withdraw anytime; withdrawal won’t affect your rights to the extent data has been anonymized.”

Anonymization / pseudonymization

• Pseudonymize before transfer: replace IDs with salted HMACs; store key separately.
• Anonymize where possible: aggregate metrics, remove face audio, blur faces, drop precise geolocation.
• For biometrics that are inherently identifying, keep strict pseudonymization and minimize retention; consider synthetic or differential-privacy noise for shared aggregates.

Data Use Agreement (DUA)

• Specify permitted purposes, prohibition on re-identification, retention and destruction timelines, security measures, breach notification timelines, audit rights, and publication/reuse rules.
• Clarify controller/processor responsibilities and cross-border transfer clauses.

Technical safeguards

• Encrypt at rest and in transit (AES-256, TLS1.2+).
• Use secure transfer (SFTP or pre-signed short-lived cloud links) and destination access limited to named researchers with MFA and role-based access.
• Logging, access reviews, and endpoint DLP.
• Provide a hashed join key for linkage only when absolutely necessary; never share mapping file.

Operational controls & governance

• Train recipients on GDPR obligations, conduct a pre-transfer audit, require yearly compliance attestations, and schedule deletion verification.