Research Design, Methodology, and Rigor Questions

Approach overview

• I’d compute per-arm sample size for detecting a clinically/practically meaningful difference vs control, then adjust for multiple comparisons and covariate adjustment.

Steps

1. Specify: primary contrast (each treatment vs control), two-sided alpha, target power (1-β), expected effect size δ (mean difference) and within-group SD σ, equal allocation.
2. Use the two-sample means formula (equal n per arm):
   n0 = 2 * (Z_{1-α*}^{ } + Z_{1-β})^2 * σ^2 / δ^2
   — where α* is per-comparison alpha after multiplicity control.
3. Choose multiplicity control: Bonferroni (α* = α / 2 for two comparisons) is conservative; use Dunnett’s test for smaller penalty when comparing multiple arms to a shared control.
4. Apply ANCOVA adjustment: effective variance ≈ σ^2 * (1 - ρ^2), where ρ is correlation between baseline measure and outcome. So n_ANCOVA ≈ n0 * (1 - ρ^2). Estimate ρ from pilot or literature.
5. Add attrition buffer (e.g., inflate by 1 / (1 - dropout_rate)).

Assumptions & sensitivity

• Assume normality or large-sample CLT, equal variances, correct δ and σ estimates. Run sensitivity analyses across plausible σ, δ, and ρ. Prefer Dunnett for power-efficient multiple comparisons.

Example

• If δ = 3 units, σ = 8, α = 0.05, power = 0.8, two comparisons: Bonferroni α* = 0.025 → compute Zs, get n0 ≈ 2*(1.96+0.84)^2*64/9 ≈ 200 per arm. If ρ = 0.4, multiply by (1 - 0.16)=0.84 → ≈168 per arm, then inflate for 10% dropout → ~187.

Brief framing (why this matters)
As a design researcher, integrating hand-coded themes with telemetry lets you link rich user motivations to measurable behavior—critical for product decisions and prioritized interventions.

1) Coded features as covariates in multilevel models

• Approach: Add binary/ordinal theme indicators as predictors in hierarchical models (users nested in cohorts, sessions).
• Pros: Straightforward, directly estimates association between theme presence and behavioral outcomes; accommodates clustering and random effects (e.g., user-level intercepts/slopes).
• Cons: Assumes coded features are measured without error and effects are homogeneous unless interaction/random slopes added. Causal claims require strong assumptions.
• Practical notes: Center predictors, check multicollinearity, report marginal effects for interpretability.

2) Latent variable models (e.g., factor analysis / IRT / SEM)

• Approach: Model themes as latent constructs from multiple coded items or combine telemetry and codes into joint latent factors.
• Pros: Accounts for measurement error, captures underlying constructs, integrates heterogeneous indicators, allows mediation modeling.
• Cons: More complex, needs larger samples and identification, model fit decisions affect conclusions.
• Practical notes: Validate factor structure, report loadings, use Bayesian estimation for small samples.

3) Two-stage models (coding → prediction/modeling)

• Approach: First predict theme probability from text/coding (or establish code reliability), then use predicted scores in behavioral models.
• Pros: Scalable (automated coding), separates measurement from outcome modeling, flexible ML for prediction.
• Cons: Propagates prediction error if ignored; naive two-stage can bias estimates.
• Practical notes: Use multiple imputation or measurement-error correction; carry forward uncertainty (e.g., Bayesian joint models or bootstrap).

Trade-offs & interpretability

• Simpler covariate models are most interpretable for stakeholders; latent and joint models better for construct validity but require careful communication of assumptions.
• If goal is actionable insight, present effect sizes and predicted outcomes for personas, and be explicit about measurement error, sampling bias, and causal limits.

Recommendation
Start with multilevel models for clear, actionable associations; if themes are noisy or conceptually latent, move to latent/joint models and propagate uncertainty into downstream predictions.

Steps to compute Cohen’s kappa (applied to 200 snippets)

1) Build the contingency table

• Create a matrix of counts where rows = coder A categories, columns = coder B categories. Sum of all cells = 200.

2) Observed agreement (Po)

• Compute proportion of items where coders agree (sum of diagonal counts / 200).
• Formula:

Po = (sum of diagonal counts) / N

Plain English: fraction of snippets coded the same by both coders.

3) Expected agreement (Pe)

• For each category, compute marginal proportions pA_i and pB_i (row and column totals / N). Expected chance agreement is sum over categories of pA_i * pB_i.
• Formula:

Pe = sum over i of (pA_i * pB_i)

Plain English: probability both coders pick the same category by chance given their marginal tendencies.

4) Cohen’s kappa

• Formula:

kappa = (Po - Pe) / (1 - Pe)

Plain English: agreement beyond chance normalized by maximum possible beyond-chance agreement.

Example: if Po = 0.75 and Pe = 0.50, kappa = (0.75-0.50)/(1-0.50) = 0.5.

Interpretation for research quality (practical terms)

• k = 0.2: Poor/slight agreement — coding scheme unclear or coders need retraining; treat insights cautiously.
• k = 0.5: Moderate agreement — acceptable for exploratory work but refine codebook, adjudicate disagreements for high-stakes findings.
• k = 0.8: Substantial/strong agreement — reliable coding; good foundation for reporting thematic prevalence.

Limitations of kappa

• Sensitive to category prevalence and marginal distributions (paradox: high Po but low kappa).
• Only for two coders and mutually exclusive single-label categories.
• Treats all disagreements equally (unless weighted kappa used).

Alternatives for multi-label / non-binary coding

• Krippendorff’s alpha: handles any number of coders, missing data, and different data types (nominal, ordinal, interval).
• Fleiss’ kappa: multi-rater, single-label nominal.
• Weighted kappa: penalizes disagreements by severity (ordinal).
• Jaccard index / Hamming loss / macro/micro F1: for multi-label binary vectors per item.
• ICC (intraclass correlation): for continuous or ordinal scores.
• Gwet’s AC1/AC2: less sensitive to prevalence effects.

Recommendation: for design research with multi-label or overlapping themes, use Krippendorff’s alpha or Jaccard/F1 per label, complement quantitative agreement with codebook refinement and sample adjudication to ensure qualitative validity.

Overview / Goal
Align video (screen recording at 30 fps), eye-tracker CSV (ms timestamps), and server event logs (UNIX seconds) so gaze samples map to UI events and frames.

Practical steps

1. Collect canonical anchors
   • Ensure each data source includes at least one shared event: e.g., a visible “sync flash” in video and a matching eye-tracker marker or a programmatic client event sent to server. Record timezone/clock info.
2. Convert formats & units
   • Screen video → frame index and wall-clock: frame_time_ms = (frame_index / 30) * 1000.
   • Eye-tracker CSV → ms since device epoch (keep as integer).
   • Server logs → convert UNIX seconds to ms.
3. Compute offset(s)
   • Identify matching anchor timestamps across sources; compute pairwise offsets (video_time_ms - eye_time_ms, server_time_ms - eye_time_ms).
   • Apply offsets to translate all timestamps into a single reference (choose eye-tracker or video time).
4. Handle clock drift
   • Fit linear model if drift suspected: eye_time_aligned = a * eye_time_raw + b, using multiple anchors across session (least squares).
   • For non-linear drift, segment session and fit piecewise linear corrections.
5. Handle missing timestamps
   • If anchors missing, use heuristics: match obvious events (button clicks visible in video vs server events). If gaps, interpolate between nearest reliable anchors and flag low-confidence windows.
6. Output & validation
   • Produce a single aligned CSV: columns [aligned_time_ms, source, gaze_x, gaze_y, video_frame, ui_event, server_event_id, confidence].
   • Validate by overlaying gaze on video and checking event alignment visually for sample periods.

Formats & tools

• File formats: MP4 for video, CSV for eye-tracker, JSON/CSV for server logs.
• Tools: Python (pandas, numpy), ffmpeg (frame extraction), OpenCV for visual sync, Jupyter for validation.

Designer-specific notes

• Log when sync was imperfect and annotate affected segments in reports.
• Prioritize UX-relevant windows (task-critical moments) for manual verification so research insights remain trustworthy.

Situation & goal
I found a moderate usability improvement in a small lab study but recruitment likely had self-selection bias. I’d present a reproducible decision framework to decide: ship now, validate more, or ship with guardrails.

Framework (steps & criteria)

1. Assess potential for harm

• High harm (data loss, legal, safety) → do not ship; run further validation.
• Low harm (cosmetic, flow efficiency) → consider faster rollout or guarded ship.

2. Estimate confidence & effect robustness

• Report effect size with 95% CI; if CI crosses practical-significance threshold, treat as uncertain.
• Check sensitivity to bias: how might self-selection inflate effect? If plausible bias could explain effect, reduce confidence.

3. Evaluate reversibility & cost

• Reversible, low-cost change → prefer guarded ship or A/B test.
• High cost or hard-to-reverse → require stronger evidence.

4. Business & user impact

• Potential revenue/lift and user experience trade-offs; prioritize high-value wins with stronger validation.

5. Monitoring & rollback plan

• Define metrics (primary UX metric + safety/engagement/cohort signals), alert thresholds, sampling, and minimum exposure before decision.
• Plan quick rollback and retrospective analysis.

6. Stakeholder alignment

• Communicate uncertainty, proposed path, and timelines; get product/engineering buy-in.

Decision rules (examples)

• If low harm, reversible, CI excludes negligible effect → ship with guardrails (feature-flagged roll-out + A/B monitoring).
• If low harm but CI uncertain and bias plausible → run targeted validation (stratified online sample or randomized pilot).
• If high harm or hard-to-reverse → pause and run larger randomized trial.

Example outcome
For my study: moderate effect, low harm, reversible UI change, CI borderline and likely biased → recommend a phased rollout behind a feature flag as an A/B test, monitor primary UX metric plus retention and error rates, require pre-specified statistical thresholds and a 2-week safety window before full ship.