Design Researcher (Mid-Level) Interview Preparation Guide - FAANG Standards

Direct answer
A research dashboard for product and research teams should combine experiment-monitoring status, cohort-level analysis, and a lightweight query surface, explicitly trading some reproducibility for interactivity, since this audience's job is exploration and hypothesis generation rather than official reporting.

Structured elaboration

1. Key widgets: an experiment-summary panel (which experiments are running, their current sample-size progress and any risk flags), a cohort heatmap (rows are acquisition cohorts, columns are periods since acquisition, cells colored by retention or engagement rate, e.g. the March-2026 signup cohort's week-4 cell showing 38% still retained), a funnel view for the behavior under study, and segment drilldowns so a researcher can slice by any relevant dimension.
2. Refresh cadence: daily is usually sufficient for research and cohort analysis, though the experiment-summary panel benefits from more frequent updates so risk flags are caught early: "peeking" is checking a test's results before it has reached its planned sample size, which inflates the chance of a false positive the more often it's repeated, and "underpowered" means the test hasn't yet collected enough samples to reliably detect the effect it's looking for, so a result that looks flat could simply be noise rather than a genuine absence of effect.
3. Filters: date range, cohort/segment, and experiment selector, with the ability to save a specific slice for later reference.
4. User roles and permissions: research and product teams typically need broader query access than a standard business-user dashboard, but access to any individual-level or sensitive data should still be gated and logged.
5. Surfacing statistical significance and confidence intervals: every experiment metric shown should carry its confidence interval or a significance indicator alongside the point estimate, so a researcher doesn't over-interpret noise as signal.
6. Interactivity vs. reproducibility trade-off: exploratory, highly interactive tooling (ad-hoc filters, custom date ranges) makes hypothesis generation fast, but ad-hoc slices are easy to construct in ways that aren't reproducible or pre-registered; mitigate by clearly labeling ad-hoc exploratory views as such (distinct from the pre-registered experiment-readout view) so a promising ad-hoc finding is treated as a hypothesis to formally test, not a result to act on directly.

Worked example
A researcher notices, via an ad-hoc segment drilldown, that a feature's engagement lift measures +6% for mobile users (n=1,200) versus only +1% for desktop users (n=4,800); the mobile figure looks far more exciting, but its sample is a quarter the size of desktop's, so the dashboard flags that observation as exploratory (not a registered experiment metric, since it wasn't a pre-specified segment) and the team designs a follow-up, properly powered experiment on that specific segment rather than shipping based on the ad-hoc slice alone.

Trade-offs and pitfalls
The main risk of a highly interactive research dashboard is that ad-hoc slicing invites informal multiple-comparisons (checking many segments until one looks significant); visually distinguishing registered/pre-specified metrics from ad-hoc exploratory ones helps keep that risk visible to the people using the tool.

Requirements & constraints:

• Validate product‑market fit in 3 months with minimal spend; prioritize speed over perfection.
• Goal: clear go/no‑go decision for continued investment.

Research roadmap (12 weeks, parallel streams where possible)

Weeks 0–1: Plan & recruit

• Define target segment personas, JTBD (Jobs to Be Done: the underlying task or goal a customer is effectively "hiring" the product to do), success metrics (activation, retention intent, willingness-to-pay).
• Recruit: 50 interview candidates, 400 survey respondents, 200 landing‑page signups.

Weeks 1–4: Exploratory qual + rapid quantitative

1. Customer interviews (n=30–40)

• Method: 45–60m semi‑structured interviews with decision‑makers/users.
• Success criteria: ≥60% express problem pain ≥7/10 and current alternatives unsatisfactory.
• Output: validated pain statements, value hypotheses, pricing anchors.

2. Pricing + preference survey (n=400)

• Method: a pricing and preference survey using either conjoint analysis (respondents choose between bundles with different feature-and-price combinations, revealing how much they value each feature) or the van Westendorp method (asking the price that feels too cheap to trust, a bargain, expensive-but-worth-it, and too expensive to consider, to triangulate an acceptable price band), plus a usage-intent question.
• Success: ≥20% willing to pay target price and ≥40% indicate adoption intent (score ≥4/5).

Weeks 3–6: Rapid prototype tests
3) Landing page + acquisition test (n=200–500 visitors / 2–4 channels)

• Measure: 20%+ click‑to‑signup, 10%+ email-to-beta conversion.
• Use: validate channel CPC (cost per click, what you pay each time someone clicks the ad), headline resonance, and CTA (call to action, the specific instruction or button telling the visitor what to do next, e.g. "Start free trial").

4. Concierge / Wizard of Oz MVP (n=20–50 paid pilots or trials)

• Offer service/manual fulfillment to simulate product.
• Success: 50%+ trial-to-paid conversion or strong repeat usage signals over 2 weeks.

Weeks 6–10: Product pilots & metrics
5) Beta product with key features (n=50 users / 6–8 weeks)

• Instrument metrics: activation (first value within 7 days) ≥40%, 30-day retention ≥25%, NPS (Net Promoter Score, a 0-10 "how likely are you to recommend this" survey converted into a single score) ≥30.
• Collect qualitative feedback on friction and must‑have features.

6. Customer acquisition economics

• Track CAC (Customer Acquisition Cost: total sales and marketing spend divided by the number of customers acquired), LTV (Customer Lifetime Value: total revenue expected from a customer over their whole relationship with the product, projected from willingness-to-pay and retention), payback period ≤12 months target.

Weeks 10–12: Synthesis & decision

• Synthesize quantitative thresholds vs. targets.
• Go criteria (example): validated pain + pricing (survey) + acquisition channel with CAC < target + pilot retention/convert rates meeting thresholds.
• No-go signals: low willingness to pay (<10% at target price), CAC that isn't comfortably below LTV (for example, a projected CAC of $80 against an LTV of $150 is a workable ~1:1.9 ratio, but a CAC of $80 against an LTV of $60 means the business loses money on every customer acquired, a clear no-go), or poor retention (<10% at 30 days).

How findings drive decision:

• If go: prioritize roadmap 0–3 months (core retention workflows, billing, automated fulfillment), secure budget for scaled acquisition.
• If borderline: run two quick iterations (pricing or UX) for 4 weeks; re-evaluate.
• If no‑go: either pivot persona/channel or sunset with documented learnings.

Reporting cadence & governance:

• Weekly synthesis reports, demo of pilot usage every 2 weeks, final decision reviewed with execs at week 12 with data pack (interview themes, survey stats, funnel metrics, CAC/LTV model, pilot outcomes).

Trade-offs and pitfalls
Every sample size in this plan (30-40 interviews, 20-50 concierge pilots, 50 beta users) is directional, not statistically powered, so a hard cutoff like '≥60% express problem pain' or '≥20% willing to pay target price' should be read as a strong or weak signal, not a precise measurement with a margin of error attached. Treat convergence across multiple signals, interviews, pricing survey, and the concierge pilot pointing the same direction, as the real bar for go, and treat a single metric clearing its threshold by a hair as reason to look closer, not reason to greenlight alone. The concierge/Wizard of Oz results are also a ceiling, not a guarantee: manual fulfillment quality doesn't always survive being automated, so a strong concierge signal justifies building the real thing, not proof the finished automated product will perform the same.

Direct answer

Running a kickoff to align a new multi-stakeholder initiative means using the meeting to surface disagreement while it's still cheap to resolve, not just to announce a plan, and the agenda should be built around getting explicit, verbal agreement on goals and success criteria rather than assuming silence means alignment.

Structured elaboration

1. Pre-work, not a blank slate. Circulate a short document beforehand stating the proposed goal, scope, and success criteria, so the meeting is spent refining and confirming rather than presenting for the first time, which invites polite nodding rather than genuine engagement.
2. Structure the agenda around explicit checkpoints. Confirm the goal in the group's own words, walk through what's in and out of scope, agree on 2 to 3 concrete success metrics, and identify open risks or dependencies, with time reserved for disagreement at each step rather than rushing to "any questions" at the end.
3. Actively invite dissent. Ask directly who sees a problem with the plan, and specifically invite quieter participants to weigh in, since silence in a room with a strong personality present is not reliable evidence of agreement.
4. Close with explicit next steps and owners. End with who owns what by when, written down and shared immediately afterward, so the kickoff produces a durable artifact, not just a good feeling in the room.
5. Know it worked by what happens after, not during. A kickoff that "worked" shows up as people acting consistently with what was agreed in the following weeks; a kickoff that produced only polite nodding shows up as the same disagreements resurfacing later, framed as new information.

Worked example

Kicking off a six-month cross-functional initiative, rather than presenting a finished plan and asking "does this work for everyone," the facilitator poses a specific question to each function represented: "what's the one thing about this plan that would cause your team the most trouble." That question, asked directly rather than left as an open floor invitation, surfaces a real timeline conflict with another commitment from one participant who would not have volunteered it unprompted.

Trade-offs and pitfalls

A kickoff run this way takes longer and can feel less efficient than a crisp announcement meeting; the cost of that extra time is far smaller than the cost of discovering a fundamental disagreement three months into execution, which is what a purely informational kickoff risks.

Overview & legal basis

• Treat biometric data as highly sensitive; use explicit informed consent as legal basis and document controller/processor roles. Conduct a DPIA before sharing.

Data minimization

• Share only variables required for the research question (e.g., gaze coordinates + timestamps rather than raw video).
• Remove irrelevant PII (names, email) and trim recordings to essential segments.

Consent language (sample lines)

• “You consent to share anonymized/pseudonymized biometric (eye-tracking, heart-rate) and screen-recording data with University X for research on usability. Participation is voluntary; data will be stored for 2 years and then deleted. You may withdraw anytime; withdrawal won’t affect your rights to the extent data has been anonymized.”

Anonymization / pseudonymization

• Pseudonymize before transfer: replace IDs with salted HMACs; store key separately.
• Anonymize where possible: aggregate metrics, remove face audio, blur faces, drop precise geolocation.
• For biometrics that are inherently identifying, keep strict pseudonymization and minimize retention; consider synthetic or differential-privacy noise for shared aggregates.

Data Use Agreement (DUA)

• Specify permitted purposes, prohibition on re-identification, retention and destruction timelines, security measures, breach notification timelines, audit rights, and publication/reuse rules.
• Clarify controller/processor responsibilities and cross-border transfer clauses.

Technical safeguards

• Encrypt at rest and in transit (AES-256, TLS1.2+).
• Use secure transfer (SFTP or pre-signed short-lived cloud links) and destination access limited to named researchers with MFA and role-based access.
• Logging, access reviews, and endpoint DLP.
• Provide a hashed join key for linkage only when absolutely necessary; never share mapping file.

Operational controls & governance

• Train recipients on GDPR obligations, conduct a pre-transfer audit, require yearly compliance attestations, and schedule deletion verification.

Context & Goal
As the product designer, I'll run a 90-minute remote workshop to align Product, Eng, and Design on the top 3 research insights and get commitments to next steps.

Agenda (90 mins)

• 0-5m: Welcome & objectives (Facilitator: Product Designer)
• 5-20m: Quick research recap. 3 candidate insights presented (5m each) (Presenter: Researcher)
• 20-35m: Clarify & questions (open Q&A, scribe captures clarifications) (Scribe: PM)
• 35-55m: Breakouts (3 mixed-role groups). Map impact and risk per insight (20m)
  • Activity: 2 rounds of 10m: discuss, and vote internally on one insight priority
• 55-65m: Reconvene. Each group reports its top insight and rationale (10m) (Scribe records)
• 65-80m: Dot-vote on the top 3 across the whole room (15m)
  • Voting: each person allocates all 3 of their votes across the insights, using 2 votes as a "high priority" tag and 1 vote as a "medium priority" tag (so putting 2 votes on Insight A and 1 on Insight B makes A your top pick and B your second).
• 80-88m: Decision & commitment. Assign owners, deadlines, and first deliverables (8m)
• 88-90m: Close & next steps (2m)

Roles

• Facilitator: Product Designer (timebox, synthesize)
• Scribe: PM (capture decisions, action items)
• Researcher: presents evidence
• Engineer rep(s): surface feasibility risks

Breakout prompts

• Who benefits (user/business)?
• Technical risk & effort (low/med/high)
• Confidence & open questions

Voting mechanics

• 3 votes per person, same rule as above: 2 as "high priority," 1 as "medium."
• Use a shared board (Miro), and reveal after voting.
• Tiebreaker: the facilitator calls for a short pros/cons discussion (2 minutes per side), then the team re-votes with 1 decisive vote per stakeholder lead.

Facilitation plan for disagreements

• Surface the disagreement, restate the positions, and ask for evidence from research or metrics.
• If values differ (risk vs. speed), map the trade-offs live on a small 2x2 grid (impact vs. effort).
• Use a single-decider escalation: the PM chooses for product trade-offs; the tech lead for critical feasibility; the facilitator enforces the timebox and documents dissenting opinions.

Outcome: ranked top 3 insights, owners, a 2-week next-step plan, and captured risks and questions for follow-up.

Direct answer

Cohort analysis groups users by something they share at a fixed point in time, most commonly the week or month they signed up, and then tracks how that group behaves over subsequent periods, so that you are comparing like with like instead of blending users at very different points in their lifecycle into one trend line. An acquisition-date cohort is the most common type (grouped by signup date), and a behavioral cohort groups by a shared action instead, such as everyone who first used a specific feature in the same week.

Structured elaboration

The reason cohort analysis exists as a distinct technique, rather than just looking at a daily or weekly trend of an overall metric, is that an aggregate trend conflates two very different things: how existing users are behaving, and how the MIX of users is changing as new people join. If a product is growing fast, an aggregate "percent of users active today" trend can look flat or even improve while every individual cohort is actually retaining worse, purely because a large influx of very recent (and therefore still highly active) signups is diluting the picture. Reporting metrics by cohort instead removes that mixing effect and lets you ask a cleaner question: for people who joined at the same time, how does their behavior change as they age?

At least three retention metrics are typically reported for a cohort: day-1 retention (the fraction still active exactly one day after joining), day-7 retention (the same at one week), and rolling retention (the fraction active at any point on or after a given day, rather than on exactly that day), each answering a slightly different question about how quickly and how durably a cohort settles into use.

A concrete business use case: an e-commerce company noticing that customers acquired through a paid-search channel show markedly worse 30-day retention than customers acquired organically, even though both channels show similar day-1 numbers, would use that cohort comparison (not a blended trend line, which would hide the channel difference) to justify shifting acquisition budget toward organic-adjacent channels or investing in a channel-specific onboarding experience for paid-search users.

Worked example

A cohort of 200 users who all signed up in the same week produced the following illustrative weekly active counts: week 0 (signup week) 200 active, week 1: 110 active, week 2: 84, week 3: 68, week 4: 58, week 5: 48, week 6: 40, week 7: 34. The retention percentage for each period is the active count divided by the original 200, giving 100%, 55%, 42%, 34%, 29%, 24%, 20%, and 17%. If a second cohort acquired one month later, in a period when a new onboarding flow shipped, showed week-1 retention of 68% instead of 55% on a comparably sized cohort, that comparison (holding cohort size and week-offset fixed) is a much stronger signal that the onboarding change helped than comparing two different weeks' overall daily-active-user numbers, which would also move for reasons unrelated to the change, such as normal week-to-week traffic variation.

Trade-offs and pitfalls

Common pitfalls include comparing cohorts of very different sizes without normalizing to percentages (a cohort of 20 users retaining "50%" is much noisier evidence than a cohort of 20,000 doing the same), treating cohort analysis as interchangeable with a simple daily trend line when the two answer different questions, and forgetting that a cohort acquired very recently has an incomplete observation window, so its later-period numbers should not yet be compared directly against an older cohort's fully-observed numbers.

Direct answer
A recurring 30-minute 1:1 works best with a light, predictable structure (a quick check-in, blockers, a skill or growth item, and a career or forward-looking question), but the real skill is protecting the last two from being crowded out by whatever operational fire is loudest that week, and shifting the balance of the agenda as the relationship matures over the quarter.

Structured elaboration

A default structure for 30 minutes

Guarding against the common failure mode

A well-known failure pattern: the 1:1 happens reliably every week, on time, with all the segments technically present, but the career and growth segments become shallow ritual ("anything on your mind for growth?" "nope, all good") while blockers quietly eat the real time. The fix isn't just having a slot on the agenda, it's asking a specific, forward-looking question each cycle rather than an open-ended one, and being willing to occasionally protect that segment even when there's a real blocker competing for the time.

Diagnosing what's actually going on, not just tracking status

Part of the value of a recurring 1:1 is using it to figure out whether a struggle you're observing is a skill gap or a mindset or behavioral issue, because the two need different responses. Someone who's struggling because they don't yet know how needs teaching and practice; someone who's struggling because of avoidance, overconfidence, or a mismatch in how they're approaching the work needs a more direct conversation about the pattern itself, not more technical instruction. A 1:1 is a good place to probe for which one you're actually looking at before assuming.

An alternative structure for hands-on technical work

For roles where the most valuable use of the time is genuinely technical, a 1:1 doesn't have to follow the career-conversation template at all. Structuring it around live debugging together, walking through a real problem with explicit hypotheses ("I think it's X, here's how we'd check") and tracking which ones got ruled out, can be a more valuable use of 30 minutes than a generic status-and-goals agenda, especially early in a relationship when trust and technical credibility are still being built.

Evolving the structure over a quarter

• Early on, more of the time typically goes to blockers and establishing trust; the person needs to know the meeting is safe and useful before career conversations will be genuine rather than performative.
• As confidence builds, the balance should shift toward growth and forward-looking conversation, and the blockers segment should shrink because there's simply less friction to clear.
• If that shift isn't happening by mid-quarter, that's itself a signal worth naming directly rather than just continuing to run the same agenda.

Worked example

Situation

Early in a mentoring relationship, our 1:1s were almost entirely blockers: real, legitimate ones, but every week's slot filled up before we got near growth or career topics.

Action

I made an explicit change: reserved the last five minutes for a specific forward-looking question every time, stated as a fixed rule rather than something to get to if there was time, and moved lower-urgency blockers to async channels so they didn't have to consume the live time by default.

Result

By partway through the quarter, the ratio had genuinely shifted: blockers took less of the time because fewer new ones were coming up, and the growth and forward-looking segments started generating real, substantive conversation instead of the same shallow "all good" answer each week.

Trade-offs & pitfalls

• Mistaking a full agenda for a working one. Hitting every segment on the template doesn't mean the 1:1 is actually working if the career and growth segments are consistently shallow.
• Applying the same generic structure to a technical, debugging-heavy role. Forcing a career-conversation template onto a context where live technical problem-solving would be more valuable wastes the time on both sides.
• Not distinguishing skill gap from mindset issue. Responding to a mindset or behavioral pattern with more technical coaching, or the reverse, burns the time without addressing what's actually going on.
• Never revisiting the structure. A rigid agenda that never evolves as the mentee matures signals the relationship isn't actually progressing, even if the meeting keeps happening.

Direct answer

I decide fast-versus-rigorous by asking two questions, not by scoring urgency: how expensive is it if we're wrong, and is the decision easily reversible later? If being wrong is cheap and reversible, I go fast. If it's expensive or hard to undo, I still find time for real evidence, even under pressure, by compressing the method rather than skipping it.

Structured elaboration

Three checks drive the call:

• Cost of being wrong: a slightly worse microcopy choice costs almost nothing; a wrong call on a privacy default or a legal disclosure can cost trust or create real liability.
• Reversibility: can we ship this, watch what happens, and quietly change it next week if it's wrong? Or does it set an expectation (a data practice, a pricing commitment) that's hard to walk back once people have adjusted to it?
• Existing signal: do we already have strong evidence from past work that generalizes here, or is this genuinely unknown territory where fast intuition is more likely to be wrong?

If the answer to cost and reversibility both point toward "low stakes," a fast, rough read is not a compromise, it's the right amount of rigor. If either points toward "expensive or hard to undo," I still compress the timeline (fewer participants, a tighter scope, faster synthesis) rather than compress the evidence to zero.

Worked example

Two real-feeling scenarios under the same time pressure:

• "Should this tooltip say 'Save' or 'Save changes'?" Being wrong here costs almost nothing, it's fully reversible in the next release, and we already have plenty of past microcopy tests suggesting plainer verbs perform better. I'd run three quick intercept conversations today and ship tomorrow: fast is the right call.
• "Should we change the default privacy setting for a health-data field from opt-in to opt-out?" Being wrong here is expensive (trust, and possibly regulatory exposure), it's not easily reversible once users have adjusted their expectations, and we have no existing signal on this specific population's preference. Even with a tight deadline, I'd push for the fastest real option, something like eight rapid remote interviews plus a same-week legal review, rather than skip evidence to hit the date.

Trade-offs & pitfalls

The failure mode isn't only "rushing something risky," it's also using "we need more rigor" as a way to slow-walk a stakeholder on something genuinely low-stakes, which burns credibility for the next time rigor really matters. When I explain the call to someone who just wants an answer now, I lead with the one-line reasoning ("the cost of getting this wrong outweighs the day we'd save going faster, and we don't have existing data to lean on here") plus a specific, faster-but-real alternative and a delivery date, rather than a vague "we need more time." A scoring rubric can dress this decision up in fake precision; the honest version is a defensible judgment call you can explain in one sentence, not a formula.

Direct answer

The null hypothesis (H0) is the "nothing changed" default you assume true until the data gives strong enough evidence to reject it. The alternative hypothesis (H1, or Ha) is the specific claim you're trying to find evidence for. A two-tailed test's alternative allows the effect to go either direction, so an increase or a decrease both count as evidence against H0. A one-tailed test's alternative commits in advance to only one direction, and only evidence in that direction can ever reject H0, however extreme the result is in the other direction, a one-tailed test treats it as not significant.

Structured elaboration

Defining the two hypotheses

• H0: the status-quo claim, typically "no difference" or "no effect," for example mu_new equals mu_old, or p_new equals p_old. It's what you'd believe by default absent evidence otherwise.
• H1 (or Ha): the claim there is reason to suspect and want the data to support. Hypothesis testing is structured to make H0 the thing you disprove, not the thing you prove; you never "accept H0," you either reject it or fail to reject it.

One-tailed versus two-tailed

A two-sided alternative is symmetric: H1 says the parameter is simply not equal to the null value, so evidence in either direction, higher or lower, counts against H0. A one-sided alternative picks a direction in advance, for example H1 says the new value is greater than the old one, and the entire significance level (say 5%) is allocated to that one tail of the distribution. That gives more power to detect an effect in that specific direction for the same sample size, at the cost of being structurally blind to an effect in the other direction.

Worked example

Testing whether a new checkout flow changes conversion rate, where p_old is the current rate and p_new is the new flow's rate:

Testing whether it changes the metric in either direction, appropriate when you'd act on either a lift or a drop, for example rolling back a regression just as readily as shipping an improvement:

$$H_0: p_{\text{new}} = p_{\text{old}}, \qquad H_1: p_{\text{new}} \neq p_{\text{old}}$$

Testing whether it increases the metric, appropriate only when a decrease and "no effect" would be handled identically, for example neither would ship:

$$H_0: p_{\text{new}} \le p_{\text{old}}, \qquad H_1: p_{\text{new}} > p_{\text{old}}$$

Which to choose, and why

Choose the two-sided test whenever a result in either direction would change the decision, which is the common case in product work: a checkout redesign that decreases conversion is just as actionable, roll it back, as one that increases it, so a one-sided test built only to detect increases could let a real, harmful regression show up as "not significant" simply because the test structurally can't reject in that direction. Choose a one-sided test only when a result in the "wrong" direction is genuinely not actionable, and that decision was made before seeing the data. For example, a pure cost-reduction change to backend infrastructure where the only actions are "doesn't hurt the metric" versus "measurably hurts it," never "significantly helped," might reasonably justify a directional test. That's a narrow case, though, and switching to one-sided after peeking at a result that's "almost significant" two-sided is a form of p-hacking, not a legitimate use of the one-sided test.

Trade-offs and pitfalls

• One-sided tests have more power for a fixed alpha and sample size, but that power gain exactly matches giving up the ability to detect the opposite effect. It isn't a free lunch, it's a bet on direction made in advance.
• Choosing the tail after seeing which direction the data leans is a common way to quietly gain extra apparent significance without disclosing it. Always fix the direction, or commit to two-sided, before collecting data, ideally in writing.
• Default to two-sided unless there is a specific, pre-committed, defensible reason not to. When a one-sided test is used, say so explicitly in the write-up along with the pre-registered direction and rationale, so a reader can judge whether the choice was made honestly.

There's a menu of lightweight ways to de-risk an ambiguous ask, and the named ones (a spike or POC, a scoped ticket built on stated assumptions, deferral, a quick prototype) aren't the whole list; techniques like a fake-door test, a Wizard-of-Oz stand-in, a small pilot, or simply looking at data you already have all belong on the same menu. The skill isn't memorizing the menu, it's matching the technique to what kind of ambiguity you actually have, and then keeping whatever you pick from quietly turning into the real build.

Match the technique to the unknown.

• If you don't know whether the question is even still open: check data you already have first, always, before building anything new. Support tickets, existing analytics, a past retrospective; this costs close to nothing and sometimes the question is already answered.
• If the unknown is demand (will anyone want this): a fake-door test, a button, link, or landing page for something that doesn't exist yet, measuring click-through, tells you demand without building the thing.
• If the unknown is the shape of the interaction, not whether people want it: a Wizard-of-Oz stand-in, a human manually doing what the automation would eventually do, tests the experience without building the automation, which is usually the expensive part.
• If the unknown is technical feasibility, can this even be built the way we're imagining: a timeboxed spike or proof of concept.
• If the ambiguity is small and low-stakes: skip the experiment entirely, write a scoped ticket on a stated assumption, get a quick nod from whoever owns the area, and move.
• If you need real usage signal at modest scale before deciding to go further: a small pilot.
• If the cost of being wrong is low and nobody is actually blocked waiting on you: defer, explicitly, rather than spending effort now.

How you know a lightweight prototype is enough, and don't need something bigger. Three signals: the decision is reversible and low blast radius if you're wrong, the disagreement is about one narrow factual question rather than a whole strategic direction, and a small number of examples or users would plausibly settle it either way. If any of those isn't true, for example the decision is expensive to undo, escalate to a bigger test rather than trusting a five-user prototype.

If it succeeds, what you hand to engineering isn't the throwaway code, it's a one-page brief: the assumption that got validated, the specific approach that worked, the known limitations the prototype deliberately skipped (auth, scale, error states), and a link to the throwaway artifact clearly labeled "not production," so engineering rebuilds the thing properly instead of hardening code that was never meant to survive contact with real load.

Keeping it bounded, in three dimensions.

• Time: a hard calendar boundary with a decision meeting already on the calendar, not "we'll know when we're done."
• Cost: a person-hour ceiling stated up front, for example one engineer for three days, 24 person-hours, and an explicit rule that production-grade requirements (auth, scaling, full error handling) are out of scope for this round.
• Scope: a written "won't do" list next to the "will do" list, and a rule that any request to expand scope becomes a separate, newly-approved ticket rather than silently absorbed into the current one.

Worked example. A PM has an ambiguous ask: would users want a saved-search alert feature. Retrospective check first: support tickets mentioning this over the last quarter are frequent but not conclusive enough to build on their own. Fake-door test: a "Get notified" button on the search results page for 2 weeks to 5% of traffic. Threshold set before launch: above 3% click-through, build it; below 1%, shelve it; between 1 and 3%, run one more cheap check. That check is Wizard-of-Oz: manually send a hand-built digest email to the people who clicked and see if they actually open and engage with a manual version before building the automated one.

A different discipline. An SRE has an ambiguous ask: would customers notice if a non-critical endpoint's data freshness degraded. Instead of building automated degradation logic, they Wizard-of-Oz it, manually holding one internal dashboard's data stale for a day and watching whether anyone notices or complains, before writing a single line of the real feature.

The trap: defaulting to the fanciest technique available, a full pilot or a real prototype, when five minutes checking data you already have would have answered the question. The opposite trap is just as real: using a spike to avoid ever writing an assumption down in a ticket and getting a quick answer, when the ambiguity was small enough that asking didn't need an experiment at all.