Research Methodology Selection and Tradeoffs Questions
Covers how to choose, justify, and execute research and analysis methods given research questions, stakeholder needs, and real world constraints such as limited time, budget, or access to users. Candidates should be able to compare qualitative methods such as interviews, usability testing, ethnography, and diary studies with quantitative methods such as surveys, analytics, split testing, and controlled experiments, and explain when and how to combine them into mixed methods designs. The topic includes core decision criteria and trade offs including generative versus evaluative goals, depth versus breadth, speed versus rigor, sample size and power considerations, cost versus validity, internal validity versus external generalizability, and short term versus longitudinal designs. Practical skills include aligning methodology to success metrics and business objectives, scoping minimal viable research designs, selecting sampling strategies and proxies, recruitment and instrumentation choices, pilot testing, estimation of sample size for quantitative work, mitigation of bias and threats to validity, documenting limitations and uncertainty, communicating and defending methodological choices to nonresearch stakeholders, and ensuring ethical and privacy safeguards and data quality in constrained or iterative studies.
MediumTechnical
35 practiced
Compare diary studies and longitudinal surveys for measuring behavior change over many months. Discuss differences in sampling strategies, measurement validity (recall bias), analytic complexity, attrition risk and mitigation, and the types of causal or descriptive insights each method is best suited to produce. Provide product examples for each.
Sample Answer
**Overview**Diary studies = high-frequency, in-situ self-report (often daily) capturing context-rich behaviors and triggers. Longitudinal surveys = periodic, larger-sample snapshots (weekly/monthly) tracking trends and self-reported states over months.**Sampling strategies**- Diary: purposeful or convenience samples, smaller (N=30–200) to trade breadth for depth; recruit high-engagement participants or target segments with known behaviors.- Longitudinal survey: probability or stratified sampling to support representativeness and subgroup comparisons; larger N (hundreds–thousands).**Measurement validity (recall bias)**- Diary: lower recall bias because entries near-event; richer contextual cues (photos, timestamps).- Longitudinal survey: higher recall bias for long reporting windows; use shorter reference periods, anchoring questions, and validated scales to reduce bias.**Analytic complexity**- Diary: hierarchical/time-series models (multilevel mixed effects, sequence clustering) to handle within-person correlations and time-of-day effects.- Longitudinal survey: panel models, growth-curve analysis, difference-in-differences for repeated measures; simpler if aggregated.**Attrition risk & mitigation**- Diary: high daily burden → gamification, flexible entry windows, micro-incentives, brief prompts, passive data fusion (sensors) to reduce friction.- Survey: lower frequency but still dropout → reminder cadences, recontact quotas, weighting for nonresponse, oversampling at baseline.**Best insights / causal vs descriptive**- Diary: descriptive, granular behavior patterns, context and triggers, temporal sequences; stronger for mechanism hypotheses but still observational.- Longitudinal survey: descriptive population trends, incidence/prevalence, and—when combined with quasi-experimental designs (staggered rollout, instrumental variables)—can support stronger causal claims.**Product examples**- Diary: mobile app onboarding study for a fitness tracker—daily logs + photos reveal friction points and habitual moments.- Longitudinal survey: quarterly satisfaction and usage panel for a banking app—tracks churn predictors and segment-level trend analysis.I’d choose diary when you need micro-level behavioral mechanics; choose longitudinal surveys when you need representative trend measurement or to power subgroup comparisons.
EasyTechnical
37 practiced
Explain what mixed-methods research is and describe three common mixed-methods designs (for example: sequential-explanatory, concurrent-triangulation, exploratory-sequential). For each design, provide a short product-research example showing why that sequence or concurrency is appropriate and what kinds of insights each stage produces.
Sample Answer
**What is mixed-methods research (brief)** Mixed-methods combines qualitative and quantitative methods in a single study to leverage strengths of both: breadth and generalizability from quantitative data, depth and meaning from qualitative data. As a design researcher I use it to validate findings and generate actionable product insights.**1) Sequential-explanatory (quant → qual)** - Use: Start with a survey/analytics to identify patterns, then interviews to explain why. - Product example: Analytics show high drop-off on onboarding steps; follow-up interviews reveal confusing wording and missing affordances. - Insights: Quant identifies where problems are; qualitative explains causes and user mental models to inform UX fixes.**2) Exploratory-sequential (qual → quant)** - Use: Start with interviews or diary studies to surface hypotheses, then test with a larger survey/experiment. - Product example: Ethnographies uncover new persona behaviors; a survey measures prevalence and segments by demographics. - Insights: Qual uncovers unknown needs; quant measures scope and prioritizes features.**3) Concurrent-triangulation (qual + quant simultaneously)** - Use: Collect both types in parallel to cross-validate findings. - Product example: Usability test (task success + think-aloud) run alongside SUS scores and NPS survey. - Insights: Triangulation increases confidence; discrepancies highlight areas for deeper follow-up.Each design maps to different questions: discovery (exploratory), explanation (sequential-explanatory), or validation (concurrent). Choose based on risk, timeline, and stakeholder needs.
MediumTechnical
39 practiced
Management asks whether a newly launched onboarding flow caused higher retention, but the team could not randomize rollout. Outline a rigorous analysis plan using observational data and quasi-experimental methods (for example: difference-in-differences, propensity-score matching, regression discontinuity if applicable). Explain required assumptions, the data you need, how you'd test assumption violations, and the tradeoffs involved.
Sample Answer
Situation and goal- We need to estimate whether a new onboarding flow caused higher retention but rollout wasn’t randomized. Goal: produce credible causal inference from observational data to inform design decisions.Analysis plan (stepwise)1. Data needed - User-level: treatment indicator (saw new flow), cohort/time of sign-up, retention outcomes (day-7, day-30), key covariates (device, channel, geography, user intent signals, past behavior if returning), timestamps of exposure, funnel steps. - Business events calendar (other experiments, marketing campaigns).2. Exploratory checks - Compare pre-treatment covariate balance and retention trends. - Plot aggregate retention over time for treated vs. control; test parallel pre-trends.3. Primary approaches (triangulate) a) Difference-in-differences (DiD) - Use untreated users before/after rollout (or geographic rollouts) to estimate average treatment effect. - Assumption: parallel trends — tested via pre-trend plots and placebo DiD using earlier windows. - If violated, adjust with covariates or move to alternate methods. b) Propensity-score matching / weighting (PSM / IPTW) - Model probability of receiving new flow using observed covariates; match or weight to balance distributions. - Assumption: no unobserved confounders (selection on observables). Test: covariate balance after weighting; check standardized mean differences. - Run outcome model with covariate adjustment and robust SEs. c) Regression discontinuity (RD) — if assignment cutoff exists - If rollout used a deterministic assignment (e.g., user IDs, signup date threshold), exploit that cutoff. - Assumption: no manipulation at cutoff; continuity of covariates around threshold — test with McCrary density test and covariate continuity. - RD gives local average treatment effect near threshold. d) Synthetic control (if treated unit = one region) - Build weighted control from other regions to match pre-treatment retention trajectory. - Good for aggregate-level rollouts.4. Robustness and sensitivity - Placebo tests: fake treatment dates, outcomes not expected to change. - Heterogeneity: segment by acquisition channel, power users vs. new users. - Sensitivity analyses: Rosenbaum bounds for unobserved confounding, alternative windows, alternative covariate sets.5. Trade-offs and interpretation - DiD: strong for time-varying causal claims but relies on parallel trends; vulnerable if rollout correlates with other changes. - PSM: flexible but only adjusts observed confounders; good when many measured covariates. - RD: high internal validity but estimates only local effect near cutoff. - Synthetic control: good for single treated units; needs long pre-period. - Combine methods and report range of estimates; be explicit about assumptions and plausible biases.Deliverable- Short memo with method comparisons, assumptions tested, main estimate plus sensitivity bounds, and design recommendations (e.g., future randomized rollout or A/B test to confirm).
MediumTechnical
42 practiced
Describe how you would instrument events and define analytics metrics for a new feature to enable causal analysis (e.g., pre/post or experiment-based). Include recommended naming conventions, essential event parameters, identity stitching strategies, primary vs secondary metrics, and how you would validate event quality and completeness before trusting analyses.
Sample Answer
**Situation & goal**I’d instrument a new feature so product, UX, and research teams can run pre/post comparisons and experiments to attribute behavior changes to the feature.**Naming conventions**- event_category: feature_{featureName} (e.g., feature_onboarding)- event_action: verb_noun (e.g., click_continue, complete_flow)- event_label: optional contextual id (e.g., variant_A)Consistent snake_case and include feature name and action.**Essential event parameters**- user_id (hashed), anon_id, session_id, timestamp, device, platform- feature_version / experiment_variant- flow_step, success (bool), error_code, duration_ms, context_page, funnel_position- metadata: locale, cohort_tag**Identity stitching**- Primary: deterministic user_id for logged-in (hashed PII)- Secondary: anon_id + device_fingerprint + first_touch timestamp for unauthenticated users- Link via login events: capture pre-login anon_id to map to user_id on auth**Metrics: primary vs secondary**- Primary (causal): task completion rate, time-to-complete, conversion rate- Secondary (diagnostic): click-through, abandonment at step, error rates, engagement depth**Validation before trusting**- Instrument smoke tests: fire events in staging and prod with known flows- Event schema checks (required fields, types) and automated alerts for schema drift- Sampling audits: compare raw logs to analytics totals, reconcile DAU/MAU- Data quality checks: dedupe, orphan events, null rates < threshold- Run replication analysis: reproduce key metric from raw events and UI telemetryI’d partner with engineering to implement and with data/experimentation to run a pilot experiment and iteratively refine.
MediumTechnical
35 practiced
You must present research findings to non-research stakeholders who prefer concise slides. Design a single-slide template that communicates: study objective, method and sample, headline findings with confidence/uncertainty, primary action recommendation, and limitations. Explain what to include in each zone of the slide and why, and how to tailor language for executives vs product teams.
Sample Answer
**Single‑Slide Template (visual layout, left→right / top→bottom)**- **Title (top, 1 line)** — clear study name + date. Why: orients stakeholders instantly.- **Objective (top-left, 1–2 lines)** — concise behavioral question and decision it informs (e.g., “Why do free-trial users churn within 7 days? — informs retention roadmap”). Why: links research to business choice.- **Method & Sample (below objective, compact icons + bullets)** - Method: remote moderated usability / survey / diary - N: 12 interviews, 300 survey responses; key demographics/segmentation (e.g., new vs returning). Why: signals evidence strength and relevance.- **Headline Findings (center, bold, 3 bullets max)** - Each bullet = one insight + metric (e.g., “Onboarding confusion: 62% drop in step 2”) - Add confidence indicator (High / Medium / Low) or p-value/CI for quantitative. Why: gives fast, evidence‑graded insights.- **Primary Action Recommendation (right, highlighted box)** - Specific change + expected impact (e.g., “Simplify step 2 copy & add progress bar — projected +10% retention”). Why: connects insight to next steps.- **Limitations & Uncertainty (bottom-left, tiny)** - Short bullets: sampling bias, timeframe, external factors. - What would reduce uncertainty (e.g., A/B test, larger sample). Why: honest risk framing enables appropriate decisions.- **Next Steps / Ask (bottom-right)** - Concrete asks: resources, stakeholders to involve, metrics to track.How to tailor language- Executives: use outcomes-first language, single-line ROI or risk, confidence label (High/Med/Low), tiny visual (metric + arrow). - Product teams: include brief method details, example quotes, and suggested experiments; preserve technical nuance.Design notes- Use visual hierarchy (big headline, colored confidence chips), keep slide scannable in <20s, one font size per zone.
Unlock Full Question Bank
Get access to hundreds of Research Methodology Selection and Tradeoffs interview questions and detailed answers.
