Research Study Design Under Constraints Questions

Ability to design a complete research study given constraints (timeline, budget, access to participants). Demonstrate understanding of how to scope research appropriately, select methodologies based on time/resource availability, and create a feasible research plan. Show ability to prioritize research questions and recommend trade-offs.

HardTechnical

48 practiced

A design-led initiative needs evidence to secure a $150k research budget for the year. Build an ROI-focused research roadmap that fits a $150k annual budget and a 2-person research team. Include prioritized projects, expected business metrics improved (e.g., retention, support cost), rough cost estimates per project, and how you will measure impact.

Sample Answer

**High-level goal & framing**Secure product impact from research investments: prioritize projects that drive retention, reduce support cost, and increase activation — measurable within 6–12 months. Team: 2 researchers (0.9 FTE each), $150k budget.

**Budget split (annual)**- People ops / overhead (salaries assumed internal) — Research ops & tools: $20k (UX tools, panel, transcription)- Participant incentives & recruitment: $35k- Mixed-method projects (see below): $90k- Contingency / stakeholder workshops & reporting: $5k

Total = $150k

**Prioritized projects (rank, cost, outcome, measurement)**1) Onboarding optimization (Quarter 1–2) — $30k - Methods: remote usability + cohort analytics A/B test - Expected impact: +5–8% activation → higher MRR and retention - Measure: activation funnel, 30-day retention, A/B significance

2) Support deflection study (Q2) — $20k - Methods: mixed qualitative interviews + task-based usability on top 5 support flows - Expected impact: 10–20% reduction in support tickets - Measure: ticket volume, average handle time, CSAT before/after

3) Churn-driver discovery (Q2–Q3) — $25k - Methods: exit interviews, quantitative segmentation, predictive analysis - Expected impact: identify top 3 actionable drivers; reduce churn 2–4% - Measure: churn rate by cohort, retention lift from targeted fixes

4) Monetization UX validation (Q3) — $10k - Methods: concept tests, pricing usability, prototype testing - Expected impact: +3–6% conversion on paid tiers - Measure: paid conversion rate, ARPU change

5) Continuous feedback panel + rapid testing (ongoing) — $5k - Methods: monthly surveys, rapid remote moderated tests - Expected impact: accelerate delivery, reduce wasted builds - Measure: time-to-insight, issue discovery rate

**How we’ll measure ROI**- Tie each experiment to a KPI (activation, retention, support cost, conversion)- Use A/B where feasible; otherwise pre/post with matched cohorts- Estimate revenue impact: KPI delta * ARR per user for activation/paid conversion; cost savings = reduced ticket volume * avg ticket cost- Quarterly stakeholder reviews with dashboards and a one-page ROI score for each project

**Why this plan**Focuses on high-leverage product levers (activation, churn, support) with measurable experiments, modest tooling/incentives, and continuous delivery from a 2-person team.

EasyTechnical

38 practiced

Describe when to choose moderated remote usability testing versus unmoderated testing when you have limited access to participants and only one researcher to run sessions. Cover task complexity, expected data quality, logistics (scheduling, tech), time per session, and deliverables for each approach.

MediumTechnical

45 practiced

You must choose which of several research questions to include in a customer interview round for a six-week project. Explain a scoring and trade-off approach that balances feasibility, potential impact, and risk to the product roadmap, and show how you would sequence interviews across weeks.

HardTechnical

37 practiced

Design a study to evaluate the impact of improved error messaging on support cost reduction and conversion in a low-traffic enterprise product. You cannot modify backend flows quickly. Propose an experimental or quasi-experimental plan, cost/benefit estimates, and metrics to show ROI to customer success and finance teams.

Sample Answer

**Study goal and constraints**Improve in-product error messaging (frontend only) to reduce support costs and increase conversion in a low-traffic enterprise product where backend flows can’t be changed quickly.

**Approach (quasi-experimental + lightweight experiment)**- Deploy client-side variants via feature flag / CDN: (A) current message, (B) improved contextual message + microcopy + clear CTAs (help, retry, contact rep).- Because traffic is low, run a stepped-wedge / staggered rollout across customer accounts (randomized by account strata: ARR tier, product usage). This creates within-account pre/post and between-account comparisons.- Complement with qualitative sessions: 8–12 moderated usability interviews and 20 short post-error surveys to validate comprehension and friction points.

**Metrics**Primary:- Support volume per error type (tickets/week per 100 accounts)- Cost per ticket (agent time * $/hr) → support cost saved- Conversion events after error (task completion rate, successful checkout or activation)Secondary:- Time-to-resolution, self-serve clicks, NPS change for affected flows, escalation rate

**Analysis**- Interrupted time series on rolled-out accounts to estimate level and slope change.- Difference-in-differences between early and late cohorts to control for time trends.- Bayesian estimation to handle low counts; report credible intervals.

**Cost/benefit estimate (example numbers)**- Baseline: 50 tickets/month for affected error; avg handling 20 min; agent cost $40/hr → cost = 50 * (20/60) * 40 = $667/month.- Target reduction: 40% → savings = $267/month.- Implementation: design + frontend flagging + copy testing = 40 hours @ $80/hr = $3,200 one-time.- Payback: $3,200 / $267 ≈ 12 months. Add revenue lift: if conversion uplift 2% on 100 ARR-relevant transactions/mo at avg deal $5k → incremental expected revenue = 0.02 *100 *5,000 = $10,000/mo (realistic enterprise numbers vary), substantially improving ROI.

**Deliverables & timeline**- 2-week audit & copy prototypes; 1-week instrumenting / flags; 8–12 week staggered rollout and monitoring; concurrent qualitative sessions.- Reporting: weekly dashboard (tickets, conversions), mid-study Bayesian update, final ROI report for Customer Success and Finance with CI on savings and revenue uplift.

**Why this works**- Frontend-only changes are low-risk and fast to iterate.- Staggered/quasi-experimental design gives causal leverage despite low traffic.- Combining quantitative and qualitative evidence provides both fiscal and user-centered arguments for scale.

HardTechnical

40 practiced

You must measure a small effect (≈3% absolute lift) in conversion, baseline 5% conversion, but traffic is low. Calculate the sample size required for a standard two-tailed power analysis at 80% power and alpha=0.05, then propose realistic alternatives (Bayesian methods, sequential testing, pooled cohorts) to make the test feasible under constraints.

Sample Answer

**Direct answer — sample size calculation**

- Setup: p1 = 0.05, p2 = 0.08 (absolute lift 3%), two‑sided α = 0.05 (z = 1.96), power 0.80 (zβ = 0.84).- Use two‑proportion formula (pooled variance):

text

p = (p1 + p2) / 2 = 0.065
n_per_group = [ (zα/2 * sqrt(2*p*(1-p)) + zβ * sqrt(p1*(1-p1) + p2*(1-p2)))^2 ] / (p2 - p1)^2

- Numeric result:

text

sqrt(2*p*(1-p)) ≈ 0.3486
sqrt(p1*(1-p1)+p2*(1-p2)) ≈ 0.3480
n_per_group ≈ (1.96*0.3486 + 0.84*0.3480)^2 / 0.03^2 ≈ 1,056
total ≈ 2,112 users

**Practical note:** at low traffic this may take weeks/months—calculate test duration by dividing required sample by daily eligible visitors.

Alternatives to make testing feasible- Bayesian A/B (sequential, no fixed N): run continuously, compute posterior probability that lift > 0 (or > some minimum effect). Stop when posterior probability crosses a decision threshold (e.g., 95%). More efficient with informative priors or hierarchical priors pooling similar experiments.- Sequential tests / alpha spending: use group sequential methods (O’Brien–Fleming or Pocock) to test intermittently while controlling Type I error; often needs fewer average samples than fixed N.- Pooled cohorts / hierarchical modeling: pool data across segments or time with mixed models to borrow strength and reduce variance; useful if multiple small experiments or cohorts are available.- Design changes to increase power: target higher‑conversion segments, use more sensitive metrics (e.g., click rate before conversion), convert to within‑subjects or matched cohorts, or increase treatment intensity to raise effect size.- Pragmatic approach: combine methods — pre-specify Bayesian stopping rules, focus on high-intent segments, and simulate operating characteristics before running live.

Recommendation (Design Researcher lens)- Estimate required duration given current traffic; if > acceptable, run a pilot with Bayesian sequential design and pre-registered decision rules, focus on a prioritized segment, and model outcomes with hierarchical priors to accelerate learning while controlling risk.

Unlock Full Question Bank

Get access to hundreds of Research Study Design Under Constraints interview questions and detailed answers.

Join thousands of developers preparing for their dream job.