Product Sense and Design Questions
Reasoning about what to build and why, for a given user and business context: reading a vague brief, generating and evaluating candidate feature or product concepts, and defending the chosen concept with structured logic (user needs, business impact, feasibility). Typical prompts are open-ended 'design a product for X' or 'improve product Y' briefs, scoping a Minimum Viable Product (what ships first and why), Jobs-to-be-Done style problem framing, picking a small set of metrics to judge a proposed concept's health, and making a single interaction-shape judgment call within the chosen concept (for example, opt-in vs. default-on, or exposing a power feature vs. keeping it hidden). Also covers anticipating what could go wrong with a proposed concept, such as adoption failure or harm to an underserved subgroup, before it ships. Assesses taste, creativity, and the ability to turn an ambiguous brief into a coherent, defensible product proposal. Out of scope: the multi-phase design process itself (ideation through validation, the double diamond), platform or technical-roadmap prioritization mechanics (RICE, ICE, MoSCoW, Cost of Delay scoring drills), system or infrastructure architecture, and narrating the candidate's own past work (STAR-style storytelling).
Using the Jobs-to-Be-Done framework, describe the primary job a personalized news feed is hired to do for users. Name three user pains, three measurable outcomes you'd track, and one concrete hypothesis for a change that addresses a key pain. How would you validate that hypothesis before and after building it?
Sample Answer
Direct answer
A strong candidate names the underlying job before jumping to metrics or a fix. Jobs-to-be-Done, or JTBD, describes what job a product is hired to do for the user, independent of any specific feature. A personalized feed's real job usually isn't "show me content," it's closer to "let me feel caught up and informed without spending time I don't have filtering through noise," and every pain, metric, and hypothesis that follows should trace back to that job.
Primary job, pains, and outcomes
Primary job: help me quickly find news that's relevant and trustworthy so I can feel informed and act on it (share, discuss, decide), without wasting time filtering out noise myself.
Three user pains: overload, where the feed surfaces too many similar or irrelevant stories and the user has to filter manually; staleness or low relevance, where stories feel generic or out of step with what the user actually cares about right now; and trust and diversity, where users worry the feed shows a narrow slice of viewpoints or content of uncertain credibility, undermining the "informed" part of the job.
Three measurable outcomes:
| Outcome | What it measures | Which pain it tracks |
|---|---|---|
| Time-to-first-meaningful-engagement | Time from opening the feed to a genuine read, save, or share, not just a scroll | Overload: faster time to value means less time spent filtering |
| Relevant-engagement rate | Share of shown stories the user reads, saves, or shares, weighted by time spent | Staleness and low relevance: directly reflects whether the feed surfaces the right stories |
| Return frequency | How often a user comes back to the feed | The job overall: a proxy for whether the feed delivers enough value to be worth the habit |
Those three are the outcomes you are trying to move. The third pain, trust and diversity, needs a fourth measure of a different kind: a guardrail, a number you are trying to hold flat rather than push up, because relevant-engagement rate can rise precisely by narrowing the feed. Without one, the metric set reports a win in exactly the case where the product got worse on a pain you named yourself. Two concrete, cheap options:
- Source-set breadth per user: the count of distinct sources a user engaged with over a rolling 28 days, reported as the median across users. Ship-blocking condition: it must not fall relative to the control group.
- Periodic trust survey: a single question ("does this feed show you a fair range of viewpoints?") sampled to a small slice of users each week, tracked as a trend rather than a one-off reading.
Pick one and pre-commit its threshold before the experiment starts, or it becomes something to argue about after the results are in.
Hypothesis and validation
One concrete hypothesis: adding a short, visible "why you're seeing this" line next to unfamiliar or non-followed sources will reduce the perceived-noise pain and raise relevant-engagement rate, because users are more willing to engage with an unfamiliar story once they understand why it appeared.
Before building: run a quick concept test with a clickable mockup or a manually curated stand-in for the real feature, with a handful of representative users, watching whether the explanation changes their willingness to engage with unfamiliar stories, and asking whether it reads as helpful or as an excuse.
After building: run a controlled experiment comparing the feed with and without the explanation, measuring relevant-engagement rate, return frequency, and the diversity guardrail over a few weeks, paired with a short survey on perceived trust and relevance, so you can tell whether any lift comes from genuinely better understanding rather than novelty.
Size that experiment before you run it rather than after. Detecting a 3 percentage point lift on a 22% baseline relevant-engagement rate, at the usual 80% power and 5% significance level, needs roughly 3,100 users per arm. A three-arm test (control plus two phrasings) therefore needs about 9,400 users, and a smaller test that comes back "no measurable lift" has told you nothing, since it could not have detected the effect you were looking for either way.
Worked example
Suppose the pre-build concept test with 10 to 12 users shows most, 8 of the 11 users tested, say the explanation makes them more willing to try an unfamiliar story, while a couple, 2 of the 11, say it reads as an excuse for showing them something they didn't want. That mixed signal is still useful: it tells you the wording matters as much as the feature itself, so the post-build experiment should test at least two phrasings, not just presence versus absence. Suppose that post-build experiment then runs for three weeks at about 3,100 users per arm: the better-performing phrasing lifts relevant-engagement rate from a baseline of 22% to 25% and return frequency from 2.1 to 2.3 sessions per week, the weaker phrasing shows no measurable lift on either metric, and median source-set breadth holds flat against control in both arms. That last part is what lets you ship: the engagement lift did not come from quietly narrowing the feed.
Trade-offs and pitfalls
Chasing relevant-engagement rate alone risks over-fitting the feed to what a user already likes, quietly recreating the trust and diversity pain even while the engagement number rises, which is exactly why the diversity guardrail has to be pre-committed with a threshold rather than added as a post-hoc explanation. A concept test with a handful of users can catch an obviously bad explanation but can't tell you the real effect size, so treat it as a filter before the real experiment, not a substitute for it. And return frequency is a slow-moving metric; don't expect it to move before the shorter-term engagement metric does, and don't panic if it lags behind.
During beta you observe that power users use an undocumented keyboard shortcut that greatly speeds workflows. The team proposes surfacing it as a toolbar button but designers worry about UI clutter. Describe how you would evaluate whether to expose the shortcut, alternatives to exposing it, and how you would measure the effect of any change.
Sample Answer
Direct answer
A strong candidate resists jumping straight to "add a toolbar button" or "don't," and instead treats this as a discoverability problem with more than two solutions. The real question is how to help more users benefit from the shortcut's speed without paying the UI-clutter cost of a permanent, always-visible button, and there are usually cheaper, more targeted options in between those extremes.
How to evaluate it
- Quantify current impact first: instrument the shortcut's usage, how many users rely on it, how much time or how many steps it actually saves, and whether its users skew toward an identifiable segment doing a specific repeated task.
- Talk to a handful of people already using it: how did they discover it, and what would have helped them find it sooner? This tells you whether the real problem is that nobody knows it exists, or that the workflow itself is clunky and the shortcut is a workaround.
- List alternatives to a permanent toolbar button, in roughly increasing order of visual cost: a contextual hint that surfaces the shortcut only when the relevant action is available; a searchable command palette, a lightweight, keyboard-driven menu of actions that scales to many features without adding a button for each one; a user-customizable toolbar so power users can add it themselves without changing the default for everyone; and, last, a permanent toolbar button, reserved for features valuable and broadly used enough to earn permanent screen space.
- This is progressive disclosure in action, a design principle where advanced or less-common options are revealed only when relevant, so the default experience stays simple for most users while power users can still reach the depth they need.
Measuring the effect
Run a test with at least three groups: no change, a low-cost option such as a hint or command palette, and the toolbar button if the team still wants to test it directly. Primary metrics: completion time or step count for the target workflow, and adoption rate of whichever surfacing method is tested. Also watch for a regression in the other direction: if a permanent button measurably slows down or confuses users who never use the shortcut, more mis-clicks, longer time to find other toolbar items, that's a real cost, not a hypothetical designer worry.
Decide before you look at the data which of two numbers you are judging on, because for an opt-in surface they behave very differently:
- The arm-level (intent-to-treat) number, everyone assigned to the arm whether or not they ever touched the new surface. This is the honest estimate of what shipping the change does to the product.
- The adopter number, only the people who actually used the new surface. This is the size of the benefit when it lands, but it is not causal on its own, because adopters self-select and were probably faster or more engaged to begin with.
Worked example
Suppose the team compares no change against a command palette that includes the shortcut as a discoverable action, tested on about 2,000 users per arm. After two weeks, roughly 15% of the palette arm (about 300 of 2,000 users) has adopted it, and it is used almost entirely by people who were not using the raw keyboard shortcut before.
Among those adopters, median task time on the target workflow falls from 45 seconds to about 24 seconds, a 47% reduction, which is the real size of the effect when someone actually finds the palette. The other 85% are unchanged, with their median still at 45 seconds.
Now roll that up to the arm, because this is the number a launch decision has to be made on: 0.85 x 45 + 0.15 x 24 = 41.9 seconds, an average improvement of about 3.1 seconds, or 7%, per assigned user. The arm's median barely moves at all and stays at roughly 45 seconds, because 85% of the arm is untouched and the 50th percentile of the arm still sits inside that untouched group. A median is the wrong summary statistic for a change only a minority opts into; report the adoption rate and the adopter effect separately, or the headline reads as "no effect" when a real 47% win exists inside a 15% slice.
That pattern argues for the low-cost option over a permanent toolbar button: it captures a large per-adopter benefit at almost no ongoing UI cost, and the way to grow the arm-level number is to raise the 15% adoption rate, not to spend permanent screen space.
Trade-offs and pitfalls
Jumping straight to exposing it as a button without measuring who benefits risks cluttering the interface for a workflow only a small minority uses. Jumping straight to keeping it hidden to protect simplicity, without testing anything, risks leaving real, measurable productivity on the table for engaged users. Reporting only the adopter number is the most common way this kind of test gets oversold, since a 47% improvement among self-selected adopters is not a 47% improvement to the product. And a shortcut used heavily by a narrow, high-value segment, say your most active daily users, might deserve visibility even at a modest cost to everyone else; the decision should weigh who the users are, not just how many there are.
Using the Jobs-to-be-Done framework, propose JTBD statements for a ride-sharing app's 'quiet ride' feature. For each JTBD, list acceptance criteria, minimal technical requirements, priority relative to other features, and an experiment to validate demand.
Sample Answer
Direct answer
A strong candidate names the underlying job in plain, functional-plus-emotional terms before proposing anything else. Jobs-to-be-Done, or JTBD, is a framework that describes what "job" a customer is hiring a product to do, centered on the underlying need rather than a feature request or a demographic label. A single "quiet ride" request usually hides two or three genuinely different jobs, and each deserves its own acceptance criteria, technical scope, and validation plan rather than one compromise design.
JTBD 1: the everyday decompression ride
"When I'm commuting after a long day, I want to arrive without having made small talk I didn't want, so I can use the ride to unwind instead of performing politeness."
- Acceptance criteria, written as numbers so a pilot result can actually fail them: riders can select a quiet-ride preference at booking; at least 80% of riders who selected it rate the ride's quietness 4 or better out of 5 in the post-ride survey; drivers acknowledge the preference at pickup on at least 90% of matched trips. The 90% acknowledgment bar is the load-bearing one, because below roughly nine trips in ten the rider learns the setting is unreliable and stops trusting it, which is a worse outcome than never having shipped it. The 80% satisfaction bar is deliberately looser, since a rider can rate a ride badly for reasons that have nothing to do with noise. Both are pre-pilot hypotheses and both get re-cut after the first market, but they get written as numbers first, because "most riders" and "the large majority" are criteria no result could fail.
- Minimal technical requirements: a preference toggle in the booking flow, a matching flag and acknowledgment prompt in the driver app, plus a survey question and analytics event to measure adherence.
- Priority: high. It's the broadest job (most riders have had an unwanted-conversation ride at some point) and the cheapest of the two to build.
- Experiment to validate demand: expose the option to a slice of riders, measure selection rate and satisfaction versus a holdout, paired with a short survey on why riders did or didn't select it.
JTBD 2: the work-mode ride
"When I need to take a call or focus during a ride, I want assurance there won't be background conversation or music, so I don't have to apologize to whoever's on the line."
- Acceptance criteria: riders can opt into a stricter no-conversation, no-music mode; at least 60% of riders who select it during weekday business hours report in the follow-up survey that they used the ride for work. 60% is the point at which the job named in the statement is genuinely the majority use rather than a story told about a feature people are selecting for some other reason, which would mean this second job is really the first one wearing a different label and does not need its own tier.
- Minimal technical requirements: an additional preference tier beyond basic quiet mode, with the preference persisted across a rider's saved trip settings.
- Priority: medium-high. A narrower audience (business commuters) but often willing to book more predictably or pay a small premium during weekday hours.
- Experiment to validate demand: a targeted beta invite to riders who frequently book weekday business-hours trips, tracking opt-in and repeat use.
Worked example
Suppose the JTBD 1 experiment runs for two weeks in one metro market, with the option exposed to a randomly chosen 5% of riders. Riders, not trips: the preference is a rider-level setting, and a rider who saw the toggle on Monday but not on Thursday would be sitting in both arms at once. The reading then has to be exposed arm against holdout arm, everyone in each arm counted, not selectors against non-selectors, and that distinction is most of the experiment. A meaningful minority actively select quiet ride when offered, for example about 18% of exposed riders choose it once shown the option, and it is tempting to compare those 18% against the riders who declined and report the gap as the feature's value: selectors rate post-ride quietness at roughly 4.6 out of 5 against about 3.9 for riders who declined, so the feature appears to be worth 0.7 of a point. That comparison measures a preference, not a treatment: riders who choose quiet ride are the ones who wanted quiet, so of course they rate quietness higher, and the riders who declined are not an unexposed group, they are exposed riders who said no. Read the right way, average post-ride quietness across the whole exposed arm comes in at about 4.0 out of 5 against 3.9 for the holdout arm (0.18 x 4.6 + 0.82 x 3.9 = 4.03, since the 82% who declined get the same ride they always got). That is a lift of roughly a tenth of a point rather than the 4.6-versus-3.9 the selector-only comparison would have shown, and it is the only version attributable to shipping the feature; the 18% selection rate is reported alongside it as its own result rather than folded into the satisfaction comparison.
The one thing this design cannot settle is the matching-side cost, and it is worth saying so rather than quietly claiming it. Driver acceptance and wait time are properties of a market's driver pool, not of an individual rider, and both arms here are served by the same pool in the same city, so any constraint the exposed arm creates is absorbed by cars that also serve the holdout. At 5% exposure with 18% opt-in, under 1% of the market's trips carry the constraint at all, far too little to move a fleet-level number in either arm. "Drivers accepted matched trips at about the same 97% rate as before" is therefore evidence that the pilot was too small to hurt anything, not evidence that the feature is free at full rollout. Answering that question needs a market-level design, matched cities or a switchback where the feature flips on and off market-wide on alternating blocks, run once rider demand is established. So the honest read of this pilot is an 18% opt-in rate and a real but modest arm-level satisfaction lift, enough to justify moving to the narrower work-mode test next, with the marketplace-liquidity question explicitly still open.
Trade-offs and pitfalls
Treating "quiet ride" as one feature instead of two distinct jobs produces a single compromise design that under-serves everyone: too strict for casual commuters, not reliable enough for someone on a work call. Enforcement is the hardest part, since a feature depending entirely on driver compliance needs a plan for drivers who don't follow it, or rider trust collapses fast. And prioritizing the narrower, more monetizable job before validating the broad one risks investing in a segment before you even know the base feature works; the emergency and safety button must also stay fully reachable and unaffected by any quiet-mode setting. The easiest way to get a false green light on an opt-in feature like this is to compare the riders who opted in against the riders who did not: that gap is mostly self-selection, and it will look impressive whether the feature works well or barely works at all.
Identify one thing about Spotify's user experience you would improve. Describe the problem, your proposed solution at a high level, and the measurable outcomes you'd use to evaluate success.
Sample Answer
Direct answer
I would improve Spotify's autoplay continuation, the feature that keeps playing similar music after
a queue or playlist ends. The problem: when someone is using music to hold a specific mood or pace
(a workout, a focus session), autoplay often continues on genre or artist similarity and lands on a
track with a mismatched tempo or energy, breaking the exact mood the listener built the playlist to
sustain. The fix: weight continuation choices by acoustic-feature closeness (tempo, energy, and
valence, the audio characteristics that describe a track's pace and mood) to the listener's
recently played tracks, not just artist or genre similarity, and give users a simple toggle between
"keep the vibe going" and "surprise me."
Structured elaboration
The underlying job to be done is "keep a mood going without having to manually queue more music,"
not "discover something new right now." Autoplay today mostly solves for the second job (finding
something plausibly related) even in moments when the user actually wants the first. The fix
targets that mismatch directly: instead of ranking continuation candidates mainly by collaborative
signal ("people who liked this track also liked that one") or shared genre and artist, weight
candidates by proximity to the rolling average tempo, energy, and valence of the last several tracks
played. A simple toggle exists because forcing one universal default trades one failure mode for
another: some listeners genuinely want autoplay to introduce them to something different, and a
system that only ever protects the current vibe would quietly undercut Spotify's other core value,
discovery.
Worked example
Run it as an A/B test: control keeps today's autoplay behavior, treatment uses acoustic-feature
weighting for continuation candidates. Illustrative results, meant to show what success would look
like rather than a claim about real data: skip rate on the first two autoplay tracks drops from 34%
in control to 24% in treatment, average listening session length after the original playlist ends
increases from 9 minutes to 13 minutes, and thumbs-down rate on autoplay-selected tracks drops from
6% to 4%. Crucially, the autoplay opt-out rate stays flat at roughly 5% in both arms, which is the
guardrail check that the fix actually worked rather than just making people turn the feature off and
manually curate instead.
Trade-offs and pitfalls
Over-optimizing for mood consistency risks making autoplay boringly repetitive, undercutting music
discovery, which is exactly why the "surprise me" toggle needs to exist rather than a single
tightened default. Acoustic-feature similarity is also only a proxy for mood: two tracks can share
tempo and energy and still feel emotionally mismatched, so this is a real, measurable improvement,
not a fully solved problem. Finally, the change needs to be checked against overall listening time
as the primary business metric; fixing the narrower mood-continuity complaint would not be worth
shipping if it quietly reduced how much people listen overall.
A subscription product's free trial converts poorly, and someone on the team suggests behavioral-economics tricks like a countdown timer and a hard-to-find cancel button to push conversions up. How would you decide which nudges are fair game and which cross into manipulation, and how would you measure whether the ones you keep actually work?
Sample Answer
Direct answer
A strong candidate separates the two tactics immediately, because they aren't the same kind of thing. A countdown timer tied to a real, disclosed deadline is a legitimate nudge, since it helps a genuinely indecisive user act on information that's true. A hard-to-find cancel button is a dark pattern, meaning a design that deliberately makes the choice that's worse for the user easier to fall into than the choice that's better for them, and it should be rejected regardless of how well it converts.
A test for fair game versus manipulation
Three checks to apply to any proposed nudge:
- Truthfulness: is the information behind the nudge real? A countdown tied to an actual trial-end date is honest; a countdown that quietly resets every time the page reloads is a lie dressed as urgency.
- Symmetry: is declining roughly as easy as accepting? Cancelling should take about as many steps and be about as visible as signing up was. A cancel flow buried several menus deep, or one that requires a phone call, fails this test even if no single step is individually deceptive.
- The disclosure test: would you be comfortable explaining exactly how the pattern works to the user it targets, or to a journalist, without it sounding like a confession? If the honest explanation of the mechanism sounds bad, it probably is.
Applied here: the countdown timer with a real trial-end date passes truthfulness and symmetry, so it's fair game as one input into a user's decision. The hard-to-find cancel button fails symmetry outright, since it's specifically designed to be harder than sign-up was, and it's exactly the kind of pattern regulators in several jurisdictions have targeted by requiring cancellation to be no harder than sign-up. Reject it, even if a quick test shows it reduces cancellations.
Worked example
The measurement question only applies to the nudge that survived the framework, so the timer is the one that actually gets tested. Suppose a team tests the truthful countdown timer against a no-timer control. Conversion is modestly higher in the timer variant, for example 24% versus 21% trial-to-paid conversion, while 60-day churn (around 8% in both arms) and support tickets look about the same as control, suggesting the timer genuinely helped indecisive users act rather than tricking anyone. That pairing is the method, not a formality: a conversion lift on its own is equally consistent with persuasion and with manipulation, and it is the flat churn and flat support volume 60 days out that tell the two apart. The timer is kept because the downstream pair held, and it would be reverted if 60-day churn had risen enough to absorb the extra 3 points of conversion.
The cancel flow does not get the same treatment, and it is worth being precise about why, because "let's just test it and see" is the compromise that sounds reasonable in the room. It already failed the symmetry check, which is a fact about the design rather than an open empirical question: the flow is by construction harder than sign-up was. Running the experiment anyway would mean shipping a pattern regulators have targeted to a real slice of paying customers for the two months it takes downstream metrics to say anything, a cost those users pay whatever the result. And the result cannot change the decision. If refund requests roughly doubled, say from about 2% to 4% of converted trials, with reviews mentioning "hard to cancel" rising alongside, you would revert; if they had not moved at all, you would still revert, because the symmetry failure is itself the finding. A test whose two possible outcomes lead to the same action is not a test, it is cover for a decision someone has already made. What you owe leadership instead is the forecast without the experiment: name the likely refund, chargeback and review-sentiment cost, note that it lands one to two months after the conversion lift shows up on the dashboard, and say plainly that the trade is being declined on the symmetry rule rather than on the numbers.
Trade-offs and pitfalls
Judging a nudge purely on trial-to-paid conversion measured in the first week is the single most common mistake here, since manipulation and legitimate persuasion can look identical on that one number; always pair it with a downstream health metric over 30 to 60 days, such as churn, refund and chargeback rate, or review sentiment. "Everyone in the industry does it" is not a defense, since regulatory and reputational risk accumulate quietly until they don't. And removing a manipulative pattern usually costs some short-term conversion, so be ready to make that trade explicitly rather than being talked out of the fix once the cost shows up on a dashboard. The last trap is treating an ethics question as an experiment: once a nudge fails the symmetry check the decision is made, and running it anyway only ships the harm to a real cohort while you wait two months for data that will not change what you do.
Unlock Full Question Bank
Get access to all 9 Product Sense and Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.