Research Design, Methodology, and Rigor Questions

Start with the usual individual-level sample size for detecting a difference in means (two-arm, two-sided):
n_indiv_per_arm_basic = [(Z_{1-α/2} + Z_{1-β})^2 * 2 σ^2] / δ^2
where Z are Normal quantiles, σ^2 is the individual-level variance, and δ is the absolute mean difference (or use standardized δ = difference/σ and set σ^2=1).

Design effect (equal cluster sizes)
In a cluster-randomized trial (equal cluster size m), correlation of outcomes within clusters (ICC = ρ) inflates required sample size by the design effect (DE):
DE = 1 + (m − 1) ρ
So adjust individual-level sample size:
n_indiv_per_arm_cluster = n_indiv_per_arm_basic * DE
Then number of clusters per arm K = n_indiv_per_arm_cluster / m (round up).

Variable cluster sizes (coefficient of variation)
When cluster sizes vary, let m̄ be mean cluster size and CV = sd(m)/m̄. A commonly used inflation factor is:
DE_var ≈ 1 + ρ [ m̄ (1 + CV^2) − 1 ]
So use DE_var in place of DE. Intuition: variability in cluster sizes increases the effective average pairwise correlation because larger clusters contribute disproportionately; CV^2 term captures that extra loss of information.

Practical estimation of ICC from limited pilot data

• One-way ANOVA (between/within mean squares): ICC_hat = (MSB − MSW) / (MSB + (k̄ − 1) MSW). Works for approximately equal k.
• Mixed-effects model (REML) is preferred for unbalanced designs: fit y_ij = μ + u_j + e_ij, estimate var(u)=σ_u^2, var(e)=σ_e^2, then ICC = σ_u^2 / (σ_u^2 + σ_e^2).
• Use bootstrap or profile likelihood to get CIs for ICC from small pilot samples.
• If pilot is tiny, supplement with ICCs from similar studies or meta-analytic averages; perform sensitivity analyses across plausible ρ (e.g., 0.01–0.10).
• Consider Bayesian shrinkage/prior for ICC when data are sparse to stabilize estimates.

Practical tips

• Report both number of clusters and individuals; trials are usually powered for number of clusters.
• If possible, inflate cluster count slightly to allow for uncertainty in ρ and dropouts.
• Present sample-size results for several ρ and CV values (sensitivity table) so stakeholders see robustness.

Start with a concise study overview: objectives, population, recruitment channel (in-app), instruments (short survey + optional audio/video or phone interviews), estimated n, duration, and timeline.

Risk assessment

• Low-risk expected for opinion surveys; list possible harms (privacy breach, emotional discomfort from sensitive questions).
• Mitigations: brief sensitive-question warning, option to skip, referral resources if distress occurs.

Consent language

• Short, plain-language consent at survey start: purpose, voluntary participation, what data is collected, duration, risks/benefits, contact for questions, and how to opt out.
• For interview opt-in: separate consent with explicit permission to record, use quotes, and clarification on anonymity.
• Use assent/parental consent procedures if minors involved (see below).

Compensation

• Specify amount/type, delivery method, and timing. Note that compensation is not contingent on completing all items (e.g., partial completion still compensated pro rata) to avoid coercion.

Data retention and deletion

• Define retention period and rationale (e.g., 3 years for analysis), deletion procedures, and how participants can request deletion.
• Describe backups and retention for published/archived aggregated results only.

De-identification measures

• Remove direct identifiers (name, email, device ID) before analysis.
• Use pseudonymization: unique study IDs; separate key stored encrypted and access-limited.
• For qualitative quotes, redact specific details that could re-identify participants.

Data access controls

• Role-based access, least privilege, encrypted storage (at rest and in transit), logging of access, and secure transfer protocols.
• State who will access raw vs. de-identified data (e.g., core research team only).

Special procedures for minors

• Prohibit recruiting under minimum age without parental consent; require parental consent + minor assent.
• Tailor consent language to reading level, minimize sensitive questions, and disallow recording for young children unless essential and explicitly consented by parent.
• Additional legal/regulatory compliance (COPPA, GDPR-Age rules) and IRB-specified safeguards.

Include appendices: recruitment text, full consent scripts, survey instrument, interview guide, data management plan, and sample de-identification workflow.

Definition
Statistical significance means the observed difference between groups is unlikely to have happened by random chance alone, assuming there truly is no effect (the null hypothesis). A p-value is the probability of seeing a result at least as extreme as what you observed, if the null hypothesis were true.

p-value = P(observed effect or more extreme | null hypothesis is true)

Intuition: a small p-value means the data would be surprising if nothing were really going on, not that the effect is large or important.

Worked Example
If a new landing page gets a p-value of 0.03 on conversion rate, that means: if the page truly had no effect, you'd see a difference this big or bigger only 3% of the time by chance. It does not tell you the probability that the new page is actually better, and it says nothing about how big the improvement is.

Common Misinterpretation
Treating p-value as 'the probability the effect is real' or 'the probability the null hypothesis is true.' It's neither. A statistically significant result with a tiny effect size (say, a 0.02% lift) can be true but practically meaningless, especially with a huge sample size.

Internal validity threats (cause-effect inference) — threat + concrete mitigation:

1. Confounding (unmeasured variable drives both treatment and outcome) — randomize assignment when possible; if not, collect confounders and use propensity score matching or regression adjustment.
2. Selection bias (systematic differences between groups) — use randomized controlled trials or pre-specify inclusion criteria; during analysis, use inverse probability weighting to adjust.
3. History effects (external events during study affect outcome) — use control group measured contemporaneously; time-series methods (difference-in-differences) to separate treatment from secular trends.
4. Maturation (participants change over time independent of treatment) — include age/time covariates, use randomized or matched controls, or shorten measurement window to minimize natural change.
5. Instrumentation (changes in measurement method over time) — standardize metrics and logging, version-control measurement code, and re-calibrate comparators; if change occurred, run sensitivity analysis excluding affected periods.

External validity threats (generalizability) — threat + concrete mitigation:

1. Selection effects (sample not representative of target users) — stratified sampling or reweight results to the target population using post-stratification weights.
2. Ecological validity (lab/experiment context differs from real-world) — run field A/B tests in production or staged rollouts, and validate with observational production data.
3. Interaction of treatment and setting (effect depends on environment) — run experiments across multiple segments/regions and include interaction terms to detect heterogeneity.
4. Temporal validity (effects change over time) — replicate experiments at different times, monitor long-term metrics, and use holdout groups for ongoing validation.
5. Population validity (effect differs across subpopulations) — pre-register subgroup analyses, ensure sufficient sample sizes for key segments, and report subgroup-specific effects with confidence intervals.

In planning, pre-register analysis, log data and metric definitions, and plan sensitivity checks to address both internal and external threats.

Framing
A surprisingly strong offline result is more often a sign of a methodology problem, most commonly data leakage, than a genuine breakthrough. The goal of a robustness check here is to rule out the boring explanations before believing the exciting one.

Checks to Run

• Feature leakage audit: go through each feature and ask whether it could only be known after the label is determined. A classic example is a 'days since last support ticket' feature in a churn model, which can leak the outcome if tickets are often filed right before churning.
• Time-based split, not random split: if the data has any time component, re-split so training data is strictly before test data in time. A random split can let future information leak backward through features like rolling averages or IDs that correlate with time.
• Group-aware split: check whether the same user, account, or session appears in both train and test. If so, the model may just be memorizing entities rather than learning generalizable patterns; re-split so all of one entity's rows land in only one side.
• Shuffle-label sanity check: retrain the same pipeline with the labels randomly shuffled. Performance should collapse to chance (AUC near 0.5); if it stays high, the pipeline is memorizing the shuffled labels rather than learning, which usually means duplicate or near-duplicate rows sit on both sides of the train/test split, or a preprocessing step (like target encoding or feature selection) was fit on the full dataset before splitting.
• Performance across slices: check AUC by cohort, geography, or time window in the test set. A model that's 95% on average but wildly uneven across slices is fragile in ways that a single aggregate number hides.

Common Pitfall
Accepting a great offline number without checking for leakage, then watching performance collapse to near-random once the model hits production traffic it has never memorized.