Research Design and Study Planning Questions

End to end planning and design of research studies to rigorously answer product, user experience, or scientific questions. Candidates should be able to translate business or product problems into clear and testable research questions and hypotheses and convert those questions into feasible and valid study plans. Core skills include selecting appropriate qualitative, quantitative, or mixed methods, defining primary outcomes and success metrics, aligning sampling strategy and inclusion and exclusion criteria, estimating sample sizes and articulating precision and power considerations, designing recruitment approaches and consent procedures, drafting interview guides survey items and measurement instruments with attention to reliability and validity, planning data collection workflows and quality controls, and outlining statistical and qualitative analysis plans and integration strategies for mixed methods. Candidates should also be able to identify potential confounds and threats to internal and external validity and propose mitigation approaches, scope studies to remain feasible under time and resource constraints, plan logistics timelines and resource allocation, pilot and iterate instruments, address ethical and regulatory requirements such as institutional review board review and data privacy, and communicate research plans limitations and actionable findings to stakeholders. Interviewers may probe trade offs among methodologies bias mitigation strategies reproducibility and documentation practices how the candidate managed scope and stakeholder expectations and how preliminary findings or stakeholder input influenced the evolution of research questions and study scope while avoiding scope creep.

HardTechnical

43 practiced

In a longitudinal outcomes study you suspect Missing Not At Random (MNAR) because participants with worsening outcomes tend to drop out. Outline a principled strategy including use of auxiliary variables, selection and pattern-mixture models, inverse probability weighting, and sensitivity analyses. Provide examples of sensitivity parameters and how you would present results showing robustness or lack thereof.

Sample Answer

Situation: You suspect MNAR in a longitudinal outcomes study because participants with worsening outcomes selectively drop out. Goal: estimate longitudinal effects while transparently assessing sensitivity to MNAR assumptions.

Principled strategy (stepwise)

1. Convene stakeholders & clarify estimand (e.g., mean difference at 12 months, intent-to-treat under hypothetical maintained-follow-up).2. Exploit auxiliary variables- Add baseline and intermediate predictors of both outcome and dropout (prior outcomes, health-service use, engagement metrics). These strengthen MAR plausibility and improve precision in MNAR models.3. Fit complementary models- Selection (shared-parameter) models: jointly model outcome trajectory (mixed-effects) and dropout hazard with correlated random effects. Interpret correlation ρ between outcome random intercept/slope and dropout as an MNAR mechanism.- Pattern-mixture models: stratify by missingness pattern or model outcome given pattern, then identify via identifying restrictions (e.g., last-observation-carried-forward offset, delta adjustments).- Inverse probability weighting (IPW): model dropout probability using rich covariates; use stabilized weights in weighted mixed models. IPW is MAR-consistent but sensitive to unobserved predictors.4. Sensitivity analyses (explicit MNAR exploration)- Delta-adjustment MI / pattern-mixture: after imputing under MAR, shift imputed outcomes by δ for those who drop out (e.g., δ = −0.2, −0.5, −1.0 SD) reflecting worse outcomes. Report how effect estimates change.- Selection-model sensitivity: vary correlation ρ or log-odds multiplier α linking outcome to dropout (e.g., odds ratio of dropout per 1-SD worse outcome = 1.5, 2.0, 3.0).- Tipping-point analysis: find δ or ρ value where inference (e.g., treatment effect loses significance or crosses clinically important threshold).5. Inference & uncertainty- Use multiple imputation with Rubin’s rules for each sensitivity scenario; bootstrap CIs for selection/shared-parameter models; account for weight variability in IPW (robust SEs).6. Presentation- Primary table: MAR-based estimate (preferred pre-specified) with covariates and diagnostics (weight distribution, overlap).- Sensitivity figure: plot treatment effect (y-axis) vs δ (x-axis) with 95% CI ribbons; annotate tipping point. Also show alternative curves for selection-model parameter ρ or OR.- Summary panel: set of scenarios (MAR, δ = −0.2, −0.5, −1.0; ρ = 0.2, 0.4) with effect, CI, p-value, and plausibility note.- Transparent appendix: model code, diagnostics, weight truncation, convergence details.

Example sensitivity parameters- Delta: shift = −0.2 to −1.0 standard deviations (clinical meaning: modest to large deterioration).- Selection OR: odds ratio per 1-SD worse outcome = 1.25, 1.5, 2.0.- Shared-parameter ρ: 0 (MAR) to 0.5 (strong MNAR).

Interpretation guidance- If estimates stable across plausible δ/ρ range, conclude robust. If inference reverses only at implausible δ/ρ values, conclude reasonably robust and report tipping point. If small plausible MNAR leads to different conclusions, be transparent: present both MAR and MNAR results, recommend cautious policy decisions or additional data collection (follow-up study, record auxiliary outcomes).

EasyTechnical

57 practiced

Compare probability sampling methods (simple random, stratified, cluster sampling) with non-probability approaches (convenience, quota, snowball). For a national customer satisfaction survey requiring representativeness, which sampling approach would you choose and why? If budget constraints force non-probability sampling, what compensations (weighting, calibration) would you plan and what limitations remain?

Sample Answer

Probability sampling:- Simple random: every unit has equal known chance; unbiased estimates and straightforward variance formulas; inefficient if population is heterogeneous or geographically dispersed.- Stratified: population divided into strata (e.g., region, age); sample drawn within each stratum. Improves precision for key subgroups and ensures representation; requires accurate frame and strata definitions.- Cluster: sampling clusters (e.g., households by postcode) then sampling within clusters (or one-stage). Cost-efficient for national surveys but increases design effect (higher variance) due to intra-cluster correlation.

Non-probability sampling:- Convenience: easiest, cheapest but biased and not generalizable.- Quota: enforces marginal quotas (age, gender) to mimic population margins but lacks random selection within quotas — bias remains.- Snowball: good for hard-to-reach groups but biased toward networks.

Recommendation for a national customer satisfaction survey: use stratified sampling (stratify by region, urban/rural, key demographics) with probability-based selection within strata. If field costs are high, combine stratification with clustered primary sampling units to reduce travel costs—account for design effects in analysis.

If forced to use non-probability sampling:- Compensations: collect rich auxiliary variables (age, gender, region, purchase history) and apply post-stratification weighting, raking/calibration to known population margins, and propensity-score adjustment (model probability of selection using an external benchmark).- Plan diagnostics: compare weighted sample distributions to external benchmarks, estimate variance with bootstrapping, and run sensitivity analyses.

Remaining limitations:- Unmeasured confounding: weighting cannot correct bias on unobserved variables.- Unknown selection mechanism: no valid design-based variance formulas; inference relies on modeling assumptions.- Potential high variance from extreme weights and limited ability to estimate small-domain estimates reliably.

Summary: probability-based stratified (possibly clustered) design is preferred. Non-probability can be made more usable with careful weighting/calibration and diagnostics, but results remain less defensible and require transparent caveats.

EasyTechnical

51 practiced

Describe the core elements you'd include in an Institutional Review Board (IRB) or ethics submission for a user research study that includes an in-app survey and optional follow-up interviews. Cover risk assessment, consent language, compensation, data retention and deletion, de-identification measures, data access controls, and special procedures if minors might be present.

Sample Answer

Start with a concise study overview: objectives, population, recruitment channel (in-app), instruments (short survey + optional audio/video or phone interviews), estimated n, duration, and timeline.

Risk assessment- Low-risk expected for opinion surveys; list possible harms (privacy breach, emotional discomfort from sensitive questions).- Mitigations: brief sensitive-question warning, option to skip, referral resources if distress occurs.

Consent language- Short, plain-language consent at survey start: purpose, voluntary participation, what data is collected, duration, risks/benefits, contact for questions, and how to opt out.- For interview opt-in: separate consent with explicit permission to record, use quotes, and clarification on anonymity.- Use assent/parental consent procedures if minors involved (see below).

Compensation- Specify amount/type, delivery method, and timing. Note that compensation is not contingent on completing all items (e.g., partial completion still compensated pro rata) to avoid coercion.

Data retention and deletion- Define retention period and rationale (e.g., 3 years for analysis), deletion procedures, and how participants can request deletion.- Describe backups and retention for published/archived aggregated results only.

De-identification measures- Remove direct identifiers (name, email, device ID) before analysis.- Use pseudonymization: unique study IDs; separate key stored encrypted and access-limited.- For qualitative quotes, redact specific details that could re-identify participants.

Data access controls- Role-based access, least privilege, encrypted storage (at rest and in transit), logging of access, and secure transfer protocols.- State who will access raw vs. de-identified data (e.g., core research team only).

Special procedures for minors- Prohibit recruiting under minimum age without parental consent; require parental consent + minor assent.- Tailor consent language to reading level, minimize sensitive questions, and disallow recording for young children unless essential and explicitly consented by parent.- Additional legal/regulatory compliance (COPPA, GDPR-Age rules) and IRB-specified safeguards.

Include appendices: recruitment text, full consent scripts, survey instrument, interview guide, data management plan, and sample de-identification workflow.

HardTechnical

50 practiced

Outline Python or R pseudocode to implement Inverse Probability of Treatment Weighting (IPTW) with stabilized weights for estimating the average treatment effect (ATE) in observational data. Include steps for fitting a propensity model, computing stabilized weights, examining overlap/positivity diagnostics (e.g., propensity histograms), truncating/extreme-weight handling, and variance estimation for weighted estimators. Provide pseudocode for plotting overlap diagnostics.

Sample Answer

To implement IPTW with stabilized weights for ATE, follow these steps: fit a propensity model, compute stabilized weights, check overlap/positivity, handle extreme weights, estimate weighted ATE and its variance (robust sandwich or bootstrap), and plot diagnostics.

1) Approach: use logistic regression (or gradient booster) for propensity score (PS), stabilized weights = P(T)/PS for treated and (1-P(T))/(1-PS) for control where P(T)=marginal treatment prevalence.

2) Pseudocode (Python-like):

python

# Fit propensity model
X, T, Y = ...  # covariates, treatment(0/1), outcome
prop_model = LogisticRegression().fit(X, T)
ps = prop_model.predict_proba(X)[:,1]  # propensity scores

# Stabilized weights
p_t = T.mean()
sw = np.where(T==1, p_t / ps, (1-p_t) / (1-ps))

# Diagnostics: propensity histograms / overlap
plt.hist(ps[T==1], bins=50, alpha=0.5, label='treated')
plt.hist(ps[T==0], bins=50, alpha=0.5, label='control')
plt.legend(); plt.xlabel('propensity score'); plt.title('Overlap')

# Check extreme weights
summary = pd.Series(sw).describe()
high_weights = np.sum(sw > sw.quantile(0.99))

# Truncation / stabilization option
lower, upper = 0.01, 0.99
ps_trunc = ps.clip(lower, upper)
sw_trunc = np.where(T==1, p_t/ps_trunc, (1-p_t)/(1-ps_trunc))

# Weighted ATE estimate (weighted difference in means)
ate = (sw * T * Y).sum()/ (sw * T).sum() - (sw * (1-T) * Y).sum()/ (sw * (1-T)).sum()

# Variance: robust sandwich for weighted estimator
# compute weighted residuals and use sandwich formula OR use bootstrap:
def weighted_mean(var, w):
    return (w*var).sum() / w.sum()

# bootstrap
B = 1000
ates = []
for b in range(B):
    idx = np.random.choice(np.arange(len(Y)), size=len(Y), replace=True)
    ateb = compute_ate(Y[idx], T[idx], X[idx])  # repeat above with same PS model or refit
    ates.append(ateb)
se = np.std(ates)

3) Key concepts:- Stabilized weights reduce variance by anchoring to marginal treatment prevalence.- Positivity/overlap violation appears as non-overlapping PS distributions or extreme weights.

4) Edge cases & handling:- PS very close to 0/1: truncate or weight caps (e.g., cap weights at 10 or 99th percentile).- Model misspecification: use richer models (GBM) and check covariate balance after weighting (standardized mean differences).

5) Alternatives: use doubly robust estimator (IPTW + outcome regression / AIPW) for more robustness.

R equivalents: use glm() for propensity, weights computed similarly; use sandwich::vcovHC or boot::boot for variance; plotting with ggplot2.

HardTechnical

39 practiced

You will test 15 secondary outcomes and several subgroup analyses in a single experiment. Draft a principled statistical strategy to control the false positive rate while preserving power for the primary outcome. Discuss approaches including hierarchical testing (gatekeeping), familywise error vs false discovery rate (FDR) control, Bayesian hierarchical models to borrow strength, and how to pre-specify and document exploratory analyses to avoid p-hacking. Also explain how you'd communicate results to non-technical stakeholders.

Sample Answer

Framework and goal- Protect the trial’s Type I error for the primary outcome (maintain alpha, preserve power), control multiplicity for 15 secondary outcomes and subgroup analyses transparently, and separate confirmatory vs exploratory inference.

Planned strategy1. Pre-specify hierarchy and families- Define Primary (single endpoint), Key Secondaries (up to 3 pre-chosen that matter for decision), Other Secondaries (remaining 12) and Subgroup analyses (pre-specified vs exploratory).- Place a gatekeeping scheme: test Primary at alpha=0.05 (two-sided). Only if Primary is significant do we proceed to Key Secondaries; use sequentially rejective Bonferroni/Holm within that family to control familywise error (FWER). Only if Key Secondaries pass do we test Other Secondaries (less critical) using FDR.

2. FWER vs FDR trade-offs- FWER (Bonferroni/Holm or closed testing) for Primary+Key Secondaries to avoid false claims that affect decisions. This preserves regulatory-style conservatism.- FDR (Benjamini-Hochberg) for the larger set of Other Secondaries to increase power while tolerating some expected proportion of false positives — acceptable for hypothesis generation.

3. Borrowing strength: Bayesian hierarchical models- For correlated continuous/rate secondary outcomes or subgroup treatment effects, fit a Bayesian hierarchical model with partial pooling. This: - Improves estimation precision and power via shrinkage. - Produces posterior probabilities (e.g., P(effect>0) or probability exceeding a minimum clinically important difference).- Use Bayesian outputs complementary to frequentist gatekeeping: maintain primary inference frequentist, use Bayesian for secondary estimation and decision support. Pre-specify priors (weakly informative), MCMC diagnostics, and thresholds for “strong evidence” (e.g., posterior probability > 0.975).

4. Subgroups- Limit number of pre-specified subgroup tests; analyze interaction terms rather than subgroup stratified tests. Use hierarchical modeling to borrow across subgroups; treat remaining subgroup analyses as exploratory and report with caution.

5. Pre-specification & documentation to avoid p-hacking- Lock the analysis plan (SAP) and statistical analysis code before unblinding: specify hierarchy, families, multiplicity procedures, models, covariates, handling of missing data, and sensitivity analyses.- Archive versions and timestamp in repository; publish SAP with trial registration or internal documentation.- Declare exploratory analyses explicitly in reports; label post-hoc findings and refrain from confirmatory language.

6. Implementation considerations- Simulate operating characteristics (type I error, power) across plausible effect sizes and correlation structures to tune thresholds and family sizes.- Provide reproducible code and pre-registered analysis scripts.

Communicating to non-technical stakeholders- Lead with the bottom line: primary result definitive (yes/no), key secondaries only tested if primary succeeds (why), and remaining outcomes are hypothesis-generating.- Use plain language: explain multiplicity as “many looks increase chance of false alarms,” and gatekeeping as “we only examine follow-ups if the main effect is real.”- Present visual summaries: forest plots of estimates with confidence/credible intervals, and a simple table indicating which tests were confirmatory vs exploratory and what error control was applied.- Quantify uncertainty: translate statistical thresholds into business-relevant metrics (expected number of false positives under FDR, probability of clinically meaningful benefit).- Recommend decisions tied to confirmatory results; suggest follow-up experiments for exploratory signals before major actions.

Why this is principled- Preserves strict control where it matters (primary decision), increases power through hierarchical testing and Bayesian pooling, and enforces transparency via pre-specification so stakeholders can trust which findings are confirmatory vs exploratory.

Unlock Full Question Bank

Get access to hundreds of Research Design and Study Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.