Research Planning and Fieldwork Questions

Definition & purpose (brief)
Recruitment quotas are predetermined counts or proportions of participant types we require for a study. As a design researcher I use them to ensure the sample matches research goals (representativeness, critical subgroups, or edge cases), avoid over-indexing on convenience participants, and make findings actionable for product decisions.

Three concrete e-commerce quotas

• Age bands: 18–24 (20%), 25–34 (35%), 35–49 (30%), 50+ (15%) — ensures generational differences in browsing and trust are captured.
• Purchase frequency: frequent buyers (monthly, 30%), occasional (every 3–6 months, 40%), first-time buyers (30%) — targets behavioral differences in intent and friction.
• Device type: mobile-only (50%), mobile+desktop (30%), desktop-only (20%) — captures interaction patterns and UI expectations.

How quotas help

• Capture diversity: force inclusion of varied demographics and behaviors so insights aren’t skewed.
• Ensure relevance: align sample composition to product user mix or priority segments so recommendations are applicable.

Interaction with segmentation & intersectionality
Quotas often layer with segmentation (e.g., age × purchase-frequency). For intersectional attributes (race, disability, income), use smaller targeted quotas or oversample key intersections to surface unique pain points, but balance feasibility and recruitment cost.

Guerrilla research

• When appropriate: Early concept validation or quick feedback on prototypes when participants and budget are limited.
• Main steps: Prepare 3–5 focused tasks/questions; go to public spaces or existing users; recruit on-the-spot (5–10 sessions); record notes or short audio; synthesize themes immediately.
• Insights yielded: Rapid pain points, first-impression reactions, feature desirability, language users use.
• Limitations: Non-representative sample, shallow depth, potential noise from environment—use as directional evidence, not definitive.

Hallway testing

• When appropriate: Quick usability checks on flows or UI changes before wider testing.
• Main steps: Bring a clickable prototype to teammates or nearby staff; give 3 realistic tasks; observe 5–8 participants; time tasks and capture failure points; iterate.
• Insights yielded: Usability blockers, confusing labels, critical task friction, estimates of task time.
• Limitations: Biased by internal participants’ domain knowledge; misses diversity of real users—best for surface UX issues.

Analytics deep dive

• When appropriate: After launch or to prioritize hypotheses when behavior data exists.
• Main steps: Define key metrics and funnels; segment by user cohort; run funnel, drop-off, and event analysis; combine with qualitative notes.
• Insights yielded: Quantified pain points, high-impact pages/steps, prioritization for investigation.
• Limitations: Shows what, not why; data quality or instrumentation gaps; needs follow-up qualitative work to explain behavior.

Situation & goal
We discovered our online satisfaction survey over‑represented active power users, biasing insights. Goal: quantify the bias, correct analyses where possible, and change future collection to be representative.

Diagnostics

• Compare sample vs population on key covariates (activity level, tenure, frequency, account type) using analytics cohort data.
• Compute standardized differences and KS tests for continuous covariates; chi‑square for categorical.
• Examine nonresponse patterns over time and by channel (email, in‑app).
• Run outcome regressions (satisfaction ~ power‑user + covariates) and propensity score models to assess confounding.

Statistical adjustments

• Post‑stratification / raking: weight sample to known population margins (activity band, region, device).
• Propensity score weighting: model probability of being sampled; inverse‑propensity weights to reduce selection bias.
• Regression adjustment / doubly robust estimator: combine covariate-adjusted regression with weights for robustness.
• Sensitivity analysis: bounding (E‑values, worst‑case) to show how strong unobserved bias must be to change conclusions.

Recruitment & design fixes for follow‑ups

• Stratified sampling by activity level with target quotas; oversample low‑activity users if needed.
• Multi‑channel recruitment (email, SMS, push, in‑product) with tailored incentives to improve reach.
• Shorten survey, add attention checks, and offer asynchronous interview slots to include less active users.
• Monitor realtime representativeness dashboard and stop/adjust recruitment when quotas deviate.

Communication to leadership

• Present concise executive summary: what bias was, how it affects key conclusions, corrected results, and remaining uncertainty.
• Show before/after estimates with confidence intervals and sensitivity bounds.
• Recommend immediate actions (apply weights to current analysis) and a timeline for study redesign.
• Offer implications for product decisions: which findings are robust vs. should be deprioritized until validated.
• Propose a follow‑up plan with metrics to track improved representativeness.

This approach balances statistical correction with practical recruitment changes and clear stakeholder communication to preserve trust in research-informed product decisions.

Brief framing
A well-formed research objective makes the study specific, measurable, and answerable. For product research (design researcher), include: Who, Behavior, Context, Outcome, Metric — each narrows scope and guides method and analysis.

Components, examples, and why they matter

• Who (target users)
  Example: “Customers who experienced a payment failure during checkout.”
  Why: Focuses recruitment and ensures findings apply to the right user segment.

• Behavior (what users do)
  Example: “Attempted to retry payment or abandon cart after a payment failure.”
  Why: Clarifies the observable action you’ll measure or probe in interviews/analytics.

• Context (when/where/under what conditions)
  Example: “Using mobile app on cellular network between 6–10pm.”
  Why: Context affects behavior; isolates environmental factors and improves validity.

• Outcome (desired change or insight)
  Example: “Understand reasons for abandonment and identify friction points to reduce failed payments.”
  Why: Directs the research goal — whether exploratory, explanatory, or evaluative.

• Metric (how success is measured)
  Example: “Reduce post-failure abandonment rate from 25% to 15% or identify top 3 root causes covering 70% of failures.”
  Why: Makes the objective measurable and guides whether quantitative or qualitative methods are needed.

Each component ensures the objective is actionable: who to recruit, what to observe, where to look, what decisions it will inform, and how to judge success.

Definitions — concise

• Purposive (non‑probability) sampling: Select participants based on specific characteristics or expertise (e.g., heavy purchasers, new users, accessibility needs). Good for deep discovery, hypothesis generation, and when you need targeted insight quickly.
• Random (probability) sampling: Every user in the target population has a known non‑zero chance of selection (e.g., stratified random sample from active user base). Enables statistical generalization and unbiased estimates of population metrics.

Which to choose
For a roadmap‑critical study that will affect millions, I would choose a probability sampling approach (with purposive components for complementary depth). Decisions at scale require representative, generalizable evidence to manage risk and justify product investment.

Concrete sampling plan

• Define population: all monthly active users in target markets (N).
• Stratify by key covariates: geography, device, tenure, account type, usage frequency.
• Determine sample size via power analysis for primary metric (e.g., detect 5% difference at 80% power).
• Within each stratum, draw random samples and invite users; oversample small but important strata (e.g., low‑incidence accessibility users) and weight results back to population.
• Add purposive substudies: 20–30 in‑depth interviews with extreme users and edge cases to explain mechanisms.

Trade‑offs

• Probability pros: generalizability, defensible to stakeholders. Cons: higher cost, slower recruitment, lower depth per participant.
• Purposive pros: faster, richer qualitative insight. Cons: biased estimates, cannot reliably project to millions.
  Combining both gives robust, actionable evidence: representative quantitative estimates plus qualitative explanations to guide design.