User Research Planning and Fieldwork Questions

Direct answer

I would build the screener in two layers: a short set of eligibility questions up front that filters for the right target user, kept short so it does not cost valid participants time or drop-off, and a set of fraud and data-quality signals that run partly during screening and partly after the session. Some fraud, like a professional survey-taker gaming your screener or the same person applying under a second identity, only shows up once you look at behavior, not at what someone typed into a form.

Structured elaboration

• Sequencing. Eligibility questions first (behavior or role questions that determine fit and can disqualify immediately), then a harder-to-game open-ended or scenario question, then logistics and demographics last, so you do not spend effort collecting details from someone who is about to be disqualified anyway.
• What makes you reject someone:
  1. Fails an explicit eligibility gate, for example does not use the product or does not hold the target role.
  2. Gives a generic, non-specific answer to an open-ended question, a common sign of a professional respondent giving a canned answer that could apply to any product (straight-lining, answering in a way that shows no real engagement with the specific question).
  3. Matches a duplicate signal, for example the same email, payment detail, device, or IP address as a previous respondent in this or a recent study.
  4. Shows an identity or location inconsistency, for example a stated location that does not match an IP-derived location, or demographic answers that contradict each other within the same screener.
  5. Completes a multi-minute screener in a few seconds, a marker of a bot or someone speed-clicking through without reading.
• Post-session and post-payment review path. Some fraud is only visible once the session happens, for example a respondent who clearly does not know the product despite passing the screener. Define an escalation path: flag the session, exclude its data from analysis even if payment has already been sent (protecting the incentive budget is not worth contaminating the dataset), and add the identifying signal to a suppression list so the same person cannot pass your screener again in a future study.
• Balancing speed against specificity. Keep only true disqualifiers mandatory; make everything else optional or branching so a long form does not cost you real target users. Track how long genuine target users actually take to complete the screener and trim questions if it is creating drop-off.

Worked example

A 4-question screener for a study on a budgeting app, targeting people who have used a named competitor app in the last 30 days:

1. Eligibility (mandatory): "Which of these apps have you used in the last 30 days?" with a list that includes the real target app, real competitors, and one decoy option (a made-up app name). Qualify only if a real expected item is checked and the decoy is NOT checked; checking the decoy is a classic sign of a rushed or professional respondent selecting everything.
2. Open-ended (harder to game): "What's the most annoying part of tracking your spending right now?" Reject vague, non-specific answers ("it's fine," "nothing really") that suggest low engagement or a scripted response.
3. Logistics (mandatory): availability window and device.
4. Demographics (optional, non-gating): collected but never used to disqualify.

Automated post-submit checks then compare email, device, and IP against a suppression list from prior studies, and flag any submission completed in under roughly 20 seconds if the screener is estimated to take about 90 seconds to complete honestly.

Trade-offs & pitfalls

A decoy question only works if it is something a genuine target user would clearly recognize as false; test it internally first so it does not accidentally trip up real participants. Trying to catch every possible fraud signal up front directly fights the speed goal, so it is worth prioritizing the two or three signals with the highest catch rate for your specific panel or platform rather than building an exhaustive screener nobody will finish.

Direct answer
Support logs and product telemetry, the passive, automatically-recorded data on what users click, complete, or abandon, are genuinely useful as a cheap, high-volume starting point for generating hypotheses. What worries me is treating them as sufficient on their own: they systematically miss certain users, the record itself can quietly corrupt over time, and behavior without a stated reason can't support a claim about motivation or cause. I'd treat them as a first pass and layer a small, targeted primary study on top before making a real decision.

Structured elaboration

Who is systematically missing: users who churned quietly before generating much telemetry, survivorship bias, meaning the data only reflects people who stuck around long enough to be recorded; users on older devices or app versions where tracking was never updated; anyone outside the primary supported locale if instrumentation was built English-first; offline or low-connectivity users whose events never sync; and, in support logs specifically, the large group who got frustrated and left without ever filing a ticket, versus the smaller, more vocal group who did.

How the record itself corrupts: instrumentation gaps happen when a team ships a feature and forgets to add or fix the tracking for it, so real usage exists with no record of it at all. Event-definition drift happens when engineers quietly redefine what an event means, say "signup complete" starts firing at a different step after a refactor, without updating a shared definition, so a chart that looks continuous is actually comparing two different things across the point where the definition changed.

Why behavior alone can't carry a motivational or causal claim: a click tells you what happened, not why, and it can't distinguish "this confused them" from "they intended exactly this and it worked." Attributing a cause to a raw behavioral pattern, "users abandon here because the button is unclear," is a guess dressed up as a finding until someone actually asks the people who left.

Worked example: layering a primary study on top
Telemetry shows a spike in a specific error-adjacent event for a segment of users. Rather than concluding what caused it, I'd trigger a short in-app micro-survey right after that event fires, catching intent at the moment it happens instead of days later from memory, then follow up with 5 to 6 contextual interviews or a reproduction of the flow in a moderated usability session with people who match that segment, specifically to test the leading hypothesis the telemetry suggested.

Trade-offs and pitfalls
The augmentation adds time and cost the raw data doesn't, which is exactly why teams are tempted to skip it under deadline pressure, and exactly when the biggest wrong-conclusion risk shows up. The instrumentation-drift risk is easy to miss because the chart still looks clean; it's worth explicitly asking an engineer whether an event's definition has changed before trusting a trend line that spans a big code change.

Direct answer

A pre-discovery research plan needs six things on it: the decision it unlocks, the research questions, the method and sample size, the timeline, the success criteria, and who has to sign off. Keep it to about one page by cutting anything a stakeholder doesn't need in order to say yes, and keep it specific by writing every section as something they can approve or reject, not a vague intention.

Structured elaboration

What goes in:

• The decision this research unlocks (what will we do differently depending on the answer)
• Research questions, split into primary (must be answered) and secondary (nice to have)
• Method and sample size (how you'll answer the questions, and how many people)
• Timeline with real dates, not "a few weeks"
• Success criteria: what result would count as a clear answer
• Who needs to approve, and what they're approving (scope, screener, budget)

How to keep it short enough that people actually read it:

• One page of decision-relevant content up front; push detail a stakeholder doesn't need to approve (the full discussion guide, screener wording, consent language) into an appendix or a linked doc.
• Bullets over paragraphs, and cut any section that doesn't change what a stakeholder will say yes or no to.
• Replace research jargon with a plain restatement, so a stakeholder outside the research team can read it in one pass.

How to keep it specific enough that people can actually approve it:

• Every research question should be paired with the decision it feeds, so a stakeholder can judge "is this worth answering" rather than "does this sound reasonable."
• Success criteria stated as a number or an explicit signal ("at least 4 of 6 primary users complete the task unaided") rather than a vague goal like "get feedback."
• Timeline as calendar dates, and sample size stated as a number, so a stakeholder can judge feasibility rather than trust it blindly.

Worked example

A one-pager for a discovery study on automated invoicing for small-business users might read:

• Decision this unlocks: whether to build automated invoicing this quarter or continue with manual workflows.
• Primary research question: will target users adopt automated invoicing, and at what price point.
• Method: 12 moderated interviews plus a screener survey of 150 respondents to size interest.
• Timeline: recruiting weeks 1 to 2, fieldwork weeks 3 to 4, synthesis and readout week 5.
• Success criteria: a clear yes or no on the adoption hypothesis, and at least three validated pain points ranked by frequency.
• Sign-off needed from: the product lead (scope) and the design lead (screener and prototype).

That's roughly 150 words a stakeholder can read start to finish and either approve or push back on with a specific objection, rather than a 10-page plan they skim and rubber-stamp.

Trade-offs and pitfalls

Cutting too much detail can make the plan so vague that a stakeholder technically approves it but disputes the scope later, once real findings show up; keep the decision and success criteria concrete even while trimming everything else. Naming a method without a sample size stops a stakeholder from judging how much confidence to expect. And skipping "who signs off" invites the plan to get re-litigated later by people who were never asked to approve it in the first place.

Before generating a single hypothesis, I'd pin down what "more revenue from free users" actually means to Sales, because that phrase can hide at least three different metrics, more free-to-paid conversions, higher spend from an add-on, or more ad revenue per free user, and each implies a different study. Only once that's fixed would I generate a short list of competing explanations and pick the one worth chasing first, given two weeks.

Pin down the metric first

This is the same move you'd make for a similarly shaped executive ask like "our signups are low, fix it": the sponsor's phrase almost never maps cleanly to one number. "Signups are low" could mean visits-to-signup conversion, invited-to-joined conversion, or signups from one channel specifically, each with a different likely cause and fix. Here, a 15 minute conversation with Sales to confirm they mean free-to-paid conversion, the most common reading and the one used below, saves you researching the wrong thing for two weeks.

Generate competing explanations, each with its own kind of evidence

1. Value gap: free users don't realize what the paid tier actually gives them.
2. Packaging mismatch: some free users would pay, but not for the tiers as currently bundled.
3. Upgrade friction: users who want to pay hit a confusing or broken path to actually doing it.

Decide what to chase first: reach times confidence times cost to check

Score each explanation on three things: reach (how much of the revenue gap this could account for if true), confidence (how much existing signal already points this way, versus a pure guess), and cost to check (how fast and cheap you can get a real answer). Favor whichever scores highest on reach and confidence while being cheapest to check, not whichever is most interesting to discuss.

For this ask, upgrade friction is usually the cheapest and fastest to check, a funnel analysis of the existing upgrade flow you likely already have data for, plus 5 to 6 quick interviews with users who started but didn't finish upgrading. Unless something in your funnel already points elsewhere, that makes it a reasonable first pick precisely because a two-week clock rewards the hypothesis you can falsify fastest, not necessarily the one you suspect matters most.

What the two-week clock forces you to cut

With a roadmap lock in two weeks, you don't get a large survey, a multi-week diary study, or a properly powered A/B test with enough participants to read a small effect confidently. What you keep: a same-day-or-next-day analytics pull to confirm or kill the cheapest hypothesis, 5 to 8 short interviews with a targeted segment, and a short, directional recommendation rather than a statistically validated one. You tell stakeholders explicitly that the two-week output is a prioritized bet with early evidence, not proof, and propose the properly powered experiment as a fast-follow once the roadmap has room for it.

Worked example

Say the funnel shows a large share of free users who click Upgrade abandon on the payment form itself, not on the pricing page, while support tickets mention confusing pricing tiers only a handful of times. That's a real, cheap-to-verify signal for upgrade friction and a weak one for packaging mismatch, so friction gets chased first; packaging goes on the list for a follow-up with more runway.

Trade-offs and pitfalls

The trap is treating reach times confidence times cost to check as an excuse to only ever test the cheap thing; if the cheap hypothesis keeps coming back inconclusive, that's itself evidence to escalate to a more expensive check rather than declaring the ask answered. Also resist letting Sales' framing narrow research to only the fastest monetization lever while ignoring that a bad upgrade experience could be actively costing retention too.

Direct answer

In 48 hours for 5 users on a small budget, the fastest reliable path is inviting from your own existing user base or a pre-vetted panel with a short, mandatory-only screener, over-recruiting to cover no-shows using an explicit ratio rather than hoping enough people show up. What you are trading away is recruiting rigor, broad demographic diversity and deep screening; what you keep is target-user relevance, since recruiting fast does not have to mean recruiting the wrong people.

Structured elaboration

• Over-recruit math. If short-notice, same-week remote invites typically see a meaningful no-show or cancellation rate, a common planning assumption is around 20 to 30%, then to net 5 completed sessions you should confirm more than 5. Confirmed needed is roughly target divided by assumed show rate, rounded up: 5 divided by 0.70 is about 7.1, so confirm 8 to comfortably clear 5 completions even if the assumption is slightly optimistic.
• Confirmation and cancellation policy. Send an immediate confirmation with the session link and a clear reschedule option, then reminders at 24 hours, 2 hours, and 30 minutes before the session. State the policy up front: full incentive for a cancellation more than 24 hours out, a reduced amount inside 24 hours, and full incentive for a genuine no-show only if they completed any pre-session task, to protect the relationship without rewarding pure no-shows.
• What survives when the screener is cut to the bone. Keep only true disqualifiers mandatory, for example the one or two eligibility questions that determine fit and basic availability; move everything else (detailed demographics, secondary preferences) to optional or drop it entirely for this round. A screener that takes 5 minutes to complete will cost you responses you do not have time to wait for.
• The same math at a bigger scale. If instead you had a fixed budget of, say, $2,000 spread across 20 completed remote sessions, that is $100 per completer before overhead. If your platform's history shows something like a 60% completion rate among confirmed sign-ups (some drop before ever starting the session), you would need to confirm roughly 20 divided by 0.60, about 34 people, to net 20 completions, and hold back a contingency reserve, for example an extra 10 to 15% of the budget, in case the actual completion rate for this specific recruit comes in lower than the historical assumption.

Worked example

Hour 0 to 2: define the one or two true eligibility disqualifiers and draft a 3-question screener. Hour 2 to 12: send invites to a targeted slice of the existing user base or panel matching those criteria, using the 8-confirmed target from the over-recruit math above. Hour 12 to 36: confirm sessions as responses come in, send the first reminder wave, and treat any no-response after 24 hours as a signal to widen the invite pool rather than wait passively. Hour 36 to 48: run sessions, holding 1 to 2 confirmed slots as a buffer in case of a late cancellation, so 5 completed sessions land inside the window even if 2 to 3 of the original 8 fall through.

Trade-offs & pitfalls

Say explicitly to stakeholders that 48-hour recruiting cannot hit demographic or segment quotas, these 5 users represent some real target users, not a representative cross-section, rather than letting that limitation go unstated. What is not up for trade-off is target-user relevance and basic informed consent; a rushed study still has to recruit real target users and cannot skip consent to save time.