User Research Planning and Fieldwork Questions

Direct answer

Without randomization, no single method earns full confidence on its own, so I would triangulate across at least three approaches (matched cohorts, difference-in-differences, and synthetic control), each resting on a different unverifiable assumption, and treat agreement across methods as the actual basis for the confidence I report to stakeholders.

What randomization buys you, and what you lose without it

A randomized test's core assumption is that random assignment makes the treatment and control groups comparable in expectation on every dimension, observed or not, so any difference you measure is attributable to the treatment. Every observational alternative instead assumes a specific, checkable substitute for that guarantee, and each substitute can fail silently:

• Matched cohorts (matching users who got the feature to similar users who did not, on things like prior activity, device, and tenure): assumes unconfoundedness, meaning the covariates (the measured traits you matched on, like prior activity level or tenure) capture everything relevant that differs between the groups. Diagnostic: check covariate balance after matching (the standardized difference between groups on each matched trait should be small) and check common support (enough comparable untreated users actually exist to match against).
• Difference-in-differences (comparing the change over time in the treated group against the change over time in a comparison group): assumes parallel trends, meaning the two groups would have moved together in the absence of the feature. Diagnostic: plot both groups' trends before the feature shipped and run a placebo test using an earlier, pre-treatment period to confirm no "effect" appears where none should exist.
• Synthetic control (building a weighted combination of untreated groups that tracks the treated group's pre-period trend): assumes that weighted combination is a valid stand-in for what the treated group would have done without the feature. Diagnostic: check the pre-treatment fit quality, and run the same method on untreated groups to see how often a similarly large "effect" shows up by chance.

Roughly in order of the confidence they earn: simple before-after comparison with no comparison group earns the least (it cannot separate the treatment from anything else that changed at the same time); matching earns more, but only protects against confounders you actually measured; difference-in-differences and synthetic control earn the most among these three, because their diagnostics (pre-trend plots, placebo tests) let you directly test the assumption rather than just hope it holds.

Worked example

Suppose 7-day retention in the treated group was 30% before the feature and 35% after (a 5 percentage point rise), and in a comparison group it was 32% before and 34% after (a 2 percentage point rise). The difference-in-differences estimate is (35% - 30%) - (34% - 32%) = 5pp - 2pp = 3 percentage points attributable to the feature, but only if the parallel-trends diagnostic on the pre-period holds; if the two groups were already diverging before the feature shipped, that 3 point estimate is not trustworthy on its own.

Communicating confidence to stakeholders

Report the point estimate with a confidence interval (a plausible range for the true effect, not just one number), and be explicit that this is an observational estimate, not a randomized one: state the assumption each method rests on in plain language, show the diagnostic that tested it, and present a small robustness table (does the estimate move much across the three methods, or stay roughly consistent). If the methods agree in direction and rough magnitude, say so and frame that agreement as the actual evidence, not the size of any single p-value. If they disagree, report the estimate as directional and low-confidence, and recommend a small randomized holdout on the next rollout wave to convert a suggestive finding into a credible one.

Direct answer

To hit plus or minus 3 percentage points at 95% confidence in each of the three markets, you need close to the same sample size per market regardless of population, roughly 1,050 to 1,070 respondents each using a conservative variance assumption, because precision depends on sample size, not on what share of the population you sampled, once the population is much larger than the sample. That means the smallest market needs a far larger share of the total sample than its share of daily active users, and once you have three market-level estimates you must weight them by true population size, not by sample size, to get one trustworthy combined number.

Structured elaboration

• Base sample size per market. For a proportion estimate with a target margin of error, MOE (the plus-or-minus range around your estimate), at a chosen confidence level expressed as a z-value (the number of standard deviations from the average that corresponds to that confidence level, 1.96 for 95% confidence), using the conservative worst-case assumption that the true proportion is near 50% (which produces the largest possible required sample):

$$n_0 = \frac{z^2 \cdot p(1-p)}{MOE^2}$$

• Finite population correction (FPC). If a market's population N is not much larger than n0, the required sample shrinks slightly, since you cannot sample more people than exist:

$$n = \frac{n_0}{1+\dfrac{n_0-1}{N}}$$

• Why the smallest market needs far more than its population share. If sample were allocated proportional to each market's daily active users instead of allocated equally, the smallest market would get a tiny fraction of the total and its own estimate would be far too imprecise to trust on its own; hitting the SAME plus or minus 3 point target in every market requires roughly equal per-market sample sizes, not population-proportional ones.
• Cluster sampling and the design effect (DEFF). If sampling individuals directly is operationally expensive (for example, if it is far easier to sample entire daily onboarding cohorts than to sample users one at a time), you can cluster by cohort, but clustering inflates the variance of your estimate by a design effect. If you estimate or assume a design effect of 1.5 based on how correlated behavior is within a cluster, you need to multiply your target sample size by 1.5 to keep the same precision; clustering is worth it only when the field-cost savings outweigh that inflation, and the design effect should come from a pilot, not a guess.
• Weighting at combination time. Since each market is sampled to roughly the same size regardless of population, a naive average across the three market estimates would give the smallest market's result far more influence than its true population share warrants; the combined number has to be weighted by each market's true share of daily active users, not by its share of the sample.

Worked example

With z = 1.96 (95% confidence) and MOE = 0.03, the base requirement is n0 = 1.96^2 x 0.25 / 0.03^2 ≈ 1,068 per market. For three markets with daily active users of roughly 5,000,000, 500,000, and 50,000 (an order of magnitude apart at each step), the finite population correction barely changes this because even the smallest market's population is about 47 times the required sample: it adjusts the requirement down to about 1,045, a difference too small to matter operationally. So the practical plan is close to 1,070 respondents per market, about 3,210 total, before adding a buffer for nonresponse.

Now consider allocation. The three markets total 5,550,000 daily active users, and the smallest market is only about 0.9% of that total. If the 3,210 total sample were instead allocated proportional to population, the smallest market would get only about 29 respondents, nowhere near the roughly 1,068 it needs for its own plus or minus 3 point estimate. That is the concrete cost of hitting equal per-market precision: the smallest market gets over 35 times its "fair" population share of the sample.

Finally, combining the three markets honestly. Suppose the three market-level abandonment estimates come back at 40%, 35%, and 50% for the large, medium, and small markets respectively. A naive unweighted average across the three market numbers gives 41.7%. But weighting each market's estimate by its true population share (about 90.1%, 9.0%, and 0.9% of the combined 5,550,000 users) gives a population-weighted combined estimate of about 39.6%, noticeably different because the naive average let the smallest market's 50% pull the number up far more than its actual population size justifies.

Trade-offs & pitfalls

If completion rates are lower than expected, oversample by the inverse of your expected response rate rather than discovering the shortfall at analysis time. The single most common mistake here is pooling all three markets' raw respondents into one dataset and reporting a simple overall proportion; because the markets were deliberately NOT sampled proportional to population, that silently overweights whichever market you oversampled relative to its true size, exactly as the naive-average example above shows. Also, the worst-case p = 0.5 assumption is conservative; if a pilot or prior data suggests the true abandonment rate is far from 50%, for example around 20%, the actual required sample drops to roughly 683 per market, so it is worth re-running this calculation with a real prior estimate before committing budget rather than defaulting to the worst case everywhere.

Direct answer

Translate the funnel observation into a precise, event-based definition tied to real timestamps, not a vague label like "people who churned," and enforce it by querying your event data for eligible users before you screen anyone, so the screener only confirms logistics, it never has to rely on someone correctly remembering which onboarding step they got stuck on.

Structured elaboration

The reasoning path, from observation to profile: analytics shows that a cohort (people who signed up recently) has an unusually high abandonment rate between two funnel steps, onboarding-start and activation (activation is your product's specific "aha" action, like completing a first real task). That's a pattern, not yet a recruiting definition. To convert it, pin down the exact events involved (signup, onboarding-start, activation) and choose a time window generous enough that "hasn't activated yet" doesn't get mistakenly counted as "dropped."

Concrete inclusion and exclusion logic:

• Include: signed up in the last 90 days, AND started onboarding, AND did not fire the activation event within 14 days of onboarding-start.
• Exclude: anyone who did activate within that window (they're not the target group), anyone already interviewed on this subject in the last 6 months, internal or test accounts, and enterprise-pilot customers, who have separate contractual research handling.

Enforcement at recruiting time: don't ask people to self-identify as having "dropped off during onboarding," memory of exactly which step someone reached is unreliable weeks later. Instead, run the inclusion and exclusion logic directly against the event database to build an eligible list first, and use a short screener only to confirm current willingness, platform, and scheduling, not to re-derive eligibility.

Quota layer, so the sample doesn't cluster on one segment: spread the eligible pool across platform (roughly matching your real traffic mix, for example 40% web, 30% iOS, 30% Android) and across stated use case (personal, work or team, evaluating for a company), so conclusions about "why onboarding fails" aren't secretly a story about one platform's bug or one persona's specific needs.

Worked example

Suppose 1,200 people signed up in the trailing 90 days, 700 of them started onboarding, and 300 of those never fired the activation event within 14 days. That 300-person list is the enforced eligible pool, built directly from the event query, not gathered by asking people at screener time. Draw a stratified sample from that list to hit the platform and use-case quotas, invite roughly 4 to 5 times more than needed per cell (a 14-day-old list has already gone a bit stale, so expect a lower response rate), and screen only to confirm scheduling and current relevance.

Trade-offs and pitfalls

Too short a "did not activate" window catches people who simply hadn't gotten around to it yet, a false positive for "dropped." Too long a window and the list goes stale, people forget details by the time you interview them. A purely event-based definition can also silently miss people who churned for reasons your event log doesn't capture at all, for example someone who bounced over pricing before any onboarding event fired; call out that limitation when you report findings.

Direct answer

I would build the screener in two layers: a short set of eligibility questions up front that filters for the right target user, kept short so it does not cost valid participants time or drop-off, and a set of fraud and data-quality signals that run partly during screening and partly after the session. Some fraud, like a professional survey-taker gaming your screener or the same person applying under a second identity, only shows up once you look at behavior, not at what someone typed into a form.

Structured elaboration

• Sequencing. Eligibility questions first (behavior or role questions that determine fit and can disqualify immediately), then a harder-to-game open-ended or scenario question, then logistics and demographics last, so you do not spend effort collecting details from someone who is about to be disqualified anyway.
• What makes you reject someone:
  1. Fails an explicit eligibility gate, for example does not use the product or does not hold the target role.
  2. Gives a generic, non-specific answer to an open-ended question, a common sign of a professional respondent giving a canned answer that could apply to any product (straight-lining, answering in a way that shows no real engagement with the specific question).
  3. Matches a duplicate signal, for example the same email, payment detail, device, or IP address as a previous respondent in this or a recent study.
  4. Shows an identity or location inconsistency, for example a stated location that does not match an IP-derived location, or demographic answers that contradict each other within the same screener.
  5. Completes a multi-minute screener in a few seconds, a marker of a bot or someone speed-clicking through without reading.
• Post-session and post-payment review path. Some fraud is only visible once the session happens, for example a respondent who clearly does not know the product despite passing the screener. Define an escalation path: flag the session, exclude its data from analysis even if payment has already been sent (protecting the incentive budget is not worth contaminating the dataset), and add the identifying signal to a suppression list so the same person cannot pass your screener again in a future study.
• Balancing speed against specificity. Keep only true disqualifiers mandatory; make everything else optional or branching so a long form does not cost you real target users. Track how long genuine target users actually take to complete the screener and trim questions if it is creating drop-off.

Worked example

A 4-question screener for a study on a budgeting app, targeting people who have used a named competitor app in the last 30 days:

1. Eligibility (mandatory): "Which of these apps have you used in the last 30 days?" with a list that includes the real target app, real competitors, and one decoy option (a made-up app name). Qualify only if a real expected item is checked and the decoy is NOT checked; checking the decoy is a classic sign of a rushed or professional respondent selecting everything.
2. Open-ended (harder to game): "What's the most annoying part of tracking your spending right now?" Reject vague, non-specific answers ("it's fine," "nothing really") that suggest low engagement or a scripted response.
3. Logistics (mandatory): availability window and device.
4. Demographics (optional, non-gating): collected but never used to disqualify.

Automated post-submit checks then compare email, device, and IP against a suppression list from prior studies, and flag any submission completed in under roughly 20 seconds if the screener is estimated to take about 90 seconds to complete honestly.

Trade-offs & pitfalls

A decoy question only works if it is something a genuine target user would clearly recognize as false; test it internally first so it does not accidentally trip up real participants. Trying to catch every possible fraud signal up front directly fights the speed goal, so it is worth prioritizing the two or three signals with the highest catch rate for your specific panel or platform rather than building an exhaustive screener nobody will finish.

The core risk to guard against is that a difference you see between languages is really a translation or cultural artifact, not a genuine usability difference, so everything below exists to reduce that risk before it ever reaches analysis.

Translate by intent, not by word
Give translators an acceptance criterion for each task rather than a literal string to translate: the goal is that the local version tests the same understanding, not that it uses the same words. For example, a task like "find and use the discount code field" should be translated so a local participant recognizes whatever field serves that purpose in that market, even if there's no local phrase that maps word for word onto "discount code." Have someone based in-country review the translated task wording before fielding, specifically checking that it lands the way you intend and doesn't carry an unintended tone or idiom.

Choosing between a local moderator and a live interpreter
A local bilingual moderator gives natural rapport and picks up on idiom and tone in real time, but needs deliberate calibration against the other languages' moderators so probing depth doesn't quietly vary by language. A researcher working through a live interpreter keeps you personally in control of the probing strategy, but adds a third-party filter that introduces a lag, can flatten emotional nuance, and tends to feel less natural to the participant. Neither is strictly better; the choice trades your control over probing against the moderator's ability to build rapport in the participant's own language.

Keeping probing depth consistent across moderators
Share one probe playbook across all moderators, with example probes and the intent behind each, not just a script to read verbatim, and spot-check a sample of sessions across languages specifically for probing depth, since a moderator who is technically following the script can still probe much less than a colleague in another market simply by asking fewer follow-ups.

Recording and transcript quality
Use professional transcription and translation with a review pass, and give transcribers a glossary of product terms in advance so the same feature is described consistently across languages rather than translated slightly differently by each transcriber.

Guarding against the central risk
Before treating any market-specific finding as a real usability difference, ask explicitly whether it could instead be explained by wording or cultural context, and check that by having the in-country reviewer look specifically at the task or question involved in that finding. Full cross-market thematic comparison, coding transcripts and comparing themes across languages, is a later analysis-phase activity and outside what this answer covers; the fieldwork-side job is making sure the sessions themselves were run comparably enough for that later comparison to mean something.

Trade-offs
An interpreter costs a beat of naturalness in exchange for keeping your own probing skill in the room; a local moderator is more natural but needs real calibration overhead, and without spot-checking for probing depth, you may not discover a moderator has been under-probing in one market until a finding from that market turns out too flat to explain anything.