Research Planning and Fieldwork Questions

Primary metric selection (single KPI)
Choose the metric that most directly maps to the business objective the CTA intends to influence. For a small text change on a CTA this is typically Click-Through Rate (CTR) if the goal is engagement, or downstream Conversion Rate (CVR) if revenue/goal completion is primary. As a Research Scientist I'd pick the metric that maximizes causal relevance to business value — e.g., CVR if conversions drive revenue; otherwise CTR.

Sensitivity & statistical design

• Estimate baseline rate and variance; compute minimum detectable effect and required sample size/power.
• Prefer metrics with lower noise (aggregated conversions) when power is limited.
• Use pre-registered A/B test with correction for multiple tests.

Guardrails (unintended consequences)

• Track secondary safety metrics: bounce rate, time on page, error rates, revenue per user, and downstream retention to detect negative trade-offs.
• Monitor heterogeneous effects across device, locale, and new vs. returning users.

Decomposition for diagnostics

• Break the primary metric into funnel stages: impressions → clicks → conversions.
• Segment by demographic, traffic source, session length.
• Time-series checks for novelty/novelty decay and early vs. steady-state effects.
  This approach aligns causal inference rigor with product impact and provides diagnostic paths if the change fails or has unexpected side effects.

Brief framing
Compare randomized A/B (gold standard) vs observational causal methods (DiD, PSM) when you cannot fully randomize an algorithmic change (e.g., rollout constrained by infra, legal, or business requirements).

A/B test — assumptions & diagnostics

• Assumptions:
  • Random assignment -> exchangeability (no unmeasured confounding).
  • Stable Unit Treatment Value Assumption (SUTVA): no interference across units.
  • Consistent measurement and no differential attrition.
• Diagnostics:
  • Pre-treatment covariate balance checks (t-tests, standardized mean differences).
  • Exposure and assignment logs to verify randomization.
  • Funnel analysis for differential dropout by arm.
• Practical note: even partial or cluster-randomized A/B with proper randomization retains causal identification.

Observational methods — assumptions & diagnostics

• Difference-in-Differences (DiD):
  • Assumption: parallel trends (treated and control would have evolved similarly absent treatment).
  • Diagnostics: pre-period trend tests, event-study coefficients, placebo interventions.
• Propensity Score Methods (PSM/IPW):
  • Assumption: conditional ignorability (no unobserved confounders given covariates) and overlap.
  • Diagnostics: covariate balance after weighting/matching, effective sample size, sensitivity analyses (Rosenbaum bounds), overlap plots.
• Practical note: check robustness to alternative covariate sets and model specifications.

When to choose which

• Prefer A/B (or cluster/encouragement designs) when feasible — fewer unverifiable assumptions.
• Use DiD when there is a clear time of change and plausible parallel trends; strengthen with event studies and multiple pre-periods.
• Use PSM/IPW when treatment assignment depends on observed covariates and you can measure rich confounders; always combine with sensitivity analysis.
• Hybrid strategies: regression discontinuity (if threshold exists), synthetic controls for single treated unit, or instrumental variables/encouragement designs when randomization incomplete.

Practical research guidance

• Prioritize designs that make assumptions testable (pre-trends, balance).
• Pre-register analysis and run blinded diagnostics.
• Report multiple methods, sensitivity bounds, and interpret causal claims with the weakest-link assumption stated explicitly.

Clarify goals & metrics

• Primary: objective task efficiency (time-to-completion, error rate), subjective user trust (validated Likert scales, behavioral proxies such as willingness to follow system suggestions).
• Secondary: cognitive load, satisfaction, retention.

Study types (triangulation)

1. Lab study — controlled tasks to measure causality: within-subjects A/B with counterbalancing, eye-tracking/think-aloud for process data.
2. Field deployment — longitudinal in-situ evaluation with real users to capture ecological validity and real-world adoption.
3. Simulation/benchmark — large-scale synthetic or logged-replay experiments to stress scalability and rare conditions.

Baseline selection

• Strong baselines: state-of-the-art automated model, existing human-in-the-loop variants, and a naive manual baseline.
• Ablations: remove key components (e.g., explanation, intervention timing) to isolate mechanisms.
• Justify each baseline with prior literature and product relevance.

Reproducibility

• Open-source code, Docker containers, seedable experiments, dataset snapshots or principled synthetic data when private (with generation code).
• Provide pre-registered analysis plan, public preprint + supplementary materials (hyperparameters, exclusion criteria, raw aggregated logs).
• Share consented anonymized user interaction logs where possible; provide synthetic equivalents otherwise.

Balancing rigor & validity

• Pre-register hypotheses and primary metrics to satisfy reviewers.
• Use lab experiments for causal claims; validate effect sizes in field for generalizability.
• Use nested design: run lab first (tight controls), then a pilot field with adaptive monitoring, then full rollout if pilot passes predefined thresholds.
• Report failures, heterogeneity analyses, and threat-to-validity section mapping trade-offs for stakeholders.

Outcome framing

• Present both statistical significance and practical impact (time saved, adoption uplift) to convince academics and product teams.

Goal: reproducible ML + UX research artifacts for conference submission and internal replication.

Checklist — What to release

• Paper-linked repository with persistent DOI (Zenodo/GitHub+Zenodo).
• Source code: training, evaluation, UI/UX study scripts, experiment orchestration.
• Data: raw, processed, and synthetic-data generator + script to reproduce synthetic dataset. If legal/privacy constraints, provide synthetic or subsampled dataset and clear access instructions.
• Pretrained model weights & checkpoints with training step metadata.
• Pre-registration documents (hypotheses, primary/secondary outcomes, analysis plan) and IRB approvals.
• Licenses, citation file, README, and quickstart.

Environment & run

• Docker image (recommended) + Dockerfile; and Conda env.yml.

FROM nvidia/cuda:11.8-runtime
RUN apt-get update && apt-get install -y git wget
COPY environment.yml /tmp/
RUN conda env create -f /tmp/environment.yml
ENTRYPOINT ["bash","-lc"]

• Repro script: ./run_all.sh that performs full pipeline: preprocess → train → eval → UX analysis → metrics aggregation.

Determinism & seeds

• Single config file listing all random seeds per stage (data split, augmentation, model init, CV folds).
• Document RNG libraries (numpy, python, torch, tensorflow) and show code to set seeds and deterministic flags.

Hardware & performance

• List exact hardware (GPU model, driver, CUDA, CPU, RAM), approximate training time and FLOPs if relevant.
• Provide low-resource reproduction options (pretrained weights + fast eval script).

Logging & provenance

• Full experiment logs (stdout, trainer logs), hyperparameter config per run, and ML metadata (wandb/MLflow run exports).
• Hashes for code commit and data files (git SHA, sha256).

Evaluation & analysis

• Evaluation scripts that compute metrics identically to paper (include exact thresholds, rounding).
• Statistical analysis notebooks used for UX studies, with anonymized raw responses and code to reproduce plots and tests.

Reviewer instructions

• Minimal reproducibility checklist in repo root with step-by-step commands to reproduce main results in ≤1 hour on recommended VM; fallback instructions for CPU-only.

Best practices & notes

• Use CI to run fast smoke tests.
• Provide checksumed artifact bundle and container registry link.
• State limitations (non-deterministic ops, proprietary data) and mitigation (seeds, synthetic proxies).

Approach summary (what I’d report and why)
I would quantify HTE uncertainty by combining stratified estimates, hierarchical Bayesian pooling, and resampling (bootstrap) to validate interval coverage—balancing bias/variance and guarding against false discoveries across many subgroups.

Stratification + plug-in CIs

• Compute subgroup ATEs and standard errors via influence-function or weighted estimators (IPW/DR) to handle assignment imbalance.
• Report Wald/Cauchy-adjusted intervals only when subgroup n is large; otherwise they are misleading.

Hierarchical Bayesian models (recommended)

• Fit a multi-level model: theta_j ~ N(mu, tau^2) with likelihoods from observed subgroup estimates or raw outcomes (Bernoulli/Gaussian/GLM).
• Posterior intervals naturally shrink noisy estimates and give coherent joint uncertainty.
• Example: use Stan to model heterogeneous slopes with partial pooling; report posterior mean, 90% and 95% credible intervals, and posterior probability that theta_j > 0.
• Caveat: priors on tau matter—use weakly informative half‑Cauchy or PC priors and check sensitivity.

Bootstrap / permutation

• Use stratified or cluster bootstrap on users to capture non-iid structure; use bootstrap percentiles or bias-corrected intervals to check frequentist coverage.
• Permutation tests provide exact null distributions for subgroup-specific nulls.

Multiplicity & low-sample pitfalls

• Multiplicity: adjust for multiple comparisons (hierarchical shrinkage, Benjamini–Hochberg for discovery lists, or Bayesian decision rules controlling posterior FDR). Report both raw and adjusted intervals/p-values.
• Low n subgroups: wide intervals; avoid over-interpretation. Consider pre-specifying subgroup analyses, pooling small groups, or hierarchical modeling rather than reporting isolated point estimates.

Practical reporting

• Provide: subgroup table with sample sizes, ATE, SE, Bayesian credible intervals, posterior probability of positive effect, and an indicator of reliability (e.g., effective sample size).
• Visuals: forest plot of posteriors plus calibration plots from bootstrap/permutation.
• Diagnostics: prior sensitivity, posterior predictive checks, and coverage checks via simulation under plausible data-generating processes.

This pipeline yields calibrated, interpretable HTE uncertainty while mitigating false discoveries and instability in small subgroups.