Product Sense and Design Questions
Reasoning about what to build and why, for a given user and business context: reading a vague brief, generating and evaluating candidate feature or product concepts, and defending the chosen concept with structured logic (user needs, business impact, feasibility). Typical prompts are open-ended 'design a product for X' or 'improve product Y' briefs, scoping a Minimum Viable Product (what ships first and why), Jobs-to-be-Done style problem framing, picking a small set of metrics to judge a proposed concept's health, and making a single interaction-shape judgment call within the chosen concept (for example, opt-in vs. default-on, or exposing a power feature vs. keeping it hidden). Also covers anticipating what could go wrong with a proposed concept, such as adoption failure or harm to an underserved subgroup, before it ships. Assesses taste, creativity, and the ability to turn an ambiguous brief into a coherent, defensible product proposal. Out of scope: the multi-phase design process itself (ideation through validation, the double diamond), platform or technical-roadmap prioritization mechanics (RICE, ICE, MoSCoW, Cost of Delay scoring drills), system or infrastructure architecture, and narrating the candidate's own past work (STAR-style storytelling).
Pick a product you personally use a lot. Describe one specific improvement you'd make to it, why it matters to users and the business, what data or evidence you'd want before committing to it, how you'd know after launch whether it worked, and how you'd guard against it backfiring for some users.
Sample Answer
Direct answer
I would improve Instacart's substitution experience: surfacing a clear, upfront likelihood that a
specific item will be substituted, based on that store's historical out-of-stock rate for it, so
shoppers can pick a backup in advance instead of being surprised by a swap after checkout. It
matters to users because unannounced substitutions are one of the most common sources of order
dissatisfaction in grocery delivery, and it matters to the business because dissatisfaction there
drives refunds, support contacts, and lower reorder rates.
Structured elaboration
Evidence I would want before committing to this: historical substitution rates broken down by
item and store, and a correlation check between orders that had an unannounced substitution and
downstream outcomes, refund requests, star ratings for that order, and whether the customer
reordered within the next two weeks. I would also want support-ticket theme analysis to confirm
substitutions are actually a top complaint category and not just an assumption from anecdotes.
How I would know after launch whether it worked: an A/B test randomized at the customer level,
not the order level, one arm sees the substitution-likelihood flag on items above a threshold, the
control arm does not. The unit is the part of this design worth defending, more than the sample
size, because two of the things I care about are not properties of an order at all. Reorder rate
within 14 days spans several orders, so an order-level split would drop a customer's first order in
treatment and their return in control and attribute the return to whichever arm the first basket
happened to land in. And cart abandonment happens before an order exists, so a cart that is
abandoned never becomes an order and could never appear in an order-level denominator at all.
Randomizing the customer and holding them in one arm for the whole window makes every session
attributable; the order-level metrics still work, they just get averaged within customer and read
with the clustering in mind, since a customer who orders weekly contributes many more orders than
one who orders monthly and the effective sample is closer to the customer count than the order
count. Primary metrics:
substitution-related refund rate and order satisfaction rating specifically for orders that
included a substitution, plus reorder rate within 14 days as a longer-horizon check that the fix
actually changed behavior, not just sentiment.
How I would guard against it backfiring: the most likely failure mode is that showing a
substitution probability scares people into removing the item or abandoning the cart entirely, which
would hurt basket size and revenue even while satisfaction improves. I would track cart-abandonment
and item-removal rate as an explicit guardrail alongside the satisfaction metrics, and only ship if
satisfaction improves without a meaningful regression there. I would also only surface the flag
above a real threshold, say a historical substitution likelihood over 40%, rather than showing a
probability on every single item; flagging everything just adds noise at exactly the moment (adding
items to a cart) when a shopper is trying to move fast.
Worked example
Suppose analysis shows orders with an unflagged substitution have an 18% refund-request rate,
versus 4% for orders with no substitution, a real, specific gap that justifies building the feature
rather than a vague sense that "substitutions are annoying." After launch, run the test on 20,000
customers per arm, each held in the same arm for the full 14-day window: suppose the treatment arm's
14-day reorder rate comes back at 62% versus 55% for control, while cart-abandonment stays flat
between the two arms. These are illustrative numbers to
show what a convincing, guardrail-clean result looks like, not a real measured outcome. The flat
abandonment number is the part that actually proves the guardrail concern didn't materialize; without
checking it, the reorder-rate win alone would not be enough to be confident the change is safe.
Both of those numbers only exist because the split is by customer. Sized as "20,000 orders per arm"
instead, the reorder rate would be measuring a return that the other arm may well have produced, and
the abandonment guardrail, the single number carrying the safety conclusion, would have no
denominator to be computed against.
Trade-offs and pitfalls
Setting the substitution-likelihood threshold too low means flagging too many items, which erodes
trust in a different direction (the app starts to feel unreliable rather than the specific product
being unreliable). The biggest pitfall is measuring only satisfaction and missing a basket-size or
revenue regression hiding behind it, since a change to a high-stakes moment like checkout can easily
improve one number while quietly damaging another. Rolling this out to 100% of users immediately,
instead of testing on a slice first, is also a real risk given how directly it touches conversion.
The subtler version of that mistake is testing on a slice split the wrong way: an order-level split
looks like a bigger sample and is easier to plumb, but it silently breaks every metric that belongs
to the customer rather than to a single basket.
Spotify wants to increase artist revenue without degrading the listener experience, and thinks an AI-powered feature could help. Propose a concrete product concept, the business assumptions behind it, and how you'd know it's actually working for both artists and listeners rather than just one side.
Sample Answer
Bottom line
Anchor the concept on a listener behavior that already exists (personal fandom for a handful of artists) rather than a promotional push, because a concept that channels money to artists as a side effect of something listeners already want to do is far less likely to feel like an ad, and both sides need pre-committed success thresholds, not one side's number used to wave away the other.
How to decide
-
Frame it with Jobs-to-be-Done (JTBD, a way of describing what a user is really trying to accomplish, independent of any specific feature): artists want a way to convert the fans who already love them into recurring support (tips, merch, tickets), not just streaming royalties, which most artists experience as too thin to matter individually. Listeners want to discover and feel closer to artists they already care about. The right concept sits at that intersection.
-
Concept: an opt-in "Direct Support" surface that appears only for a listener's small set of high-affinity artists (based on existing signals like play count concentration and saves, not a blanket promotion shown to everyone), offering a low-friction way to support that specific artist directly: a small tip, an early-access bundle, or a merch link. It never autoplays, never interrupts a session, and is easy to permanently dismiss.
-
Name the business assumptions this concept rests on, and treat each as something to test, not something to assume:
- A meaningful share of listeners already have strong, identifiable affinity for a small number of artists.
- A contextual, low-friction ask converts at a nonzero rate even though a blanket ad would not.
- The ask doesn't need to be frequent to matter in aggregate, given catalog scale.
- It doesn't simply pull revenue away from existing subscription revenue.
-
Fix the unit of analysis before writing down a single number, because this concept has two of them and they are not interchangeable. A qualifying listener is one person with at least one high-affinity artist. A listener-artist pair is one place the surface can appear, and a listener with three high-affinity artists produces three pairs. Reach and impressions are counted in pairs; retention, opt-out, and every listener-side guardrail are counted per listener. Sliding between the two is the easiest way to be off by the average number of high-affinity artists per fan without noticing.
-
Metrics have to cover both sides, since the prompt is explicit that one-sided success doesn't count:
- Artist side: incremental revenue per artist, and importantly the median lift, not just the average, since a handful of superstar artists can make an average look great while most artists in the surface see almost nothing.
- Listener side: retention and session frequency for exposed listeners versus a holdout of comparable listeners not shown the surface, plus a direct opt-out or "hide this" rate as a guardrail.
-
The actual test: randomize at the listener level, not at the pair level, and not as a simple before-and-after comparison. Each qualifying listener is either in the surface for all of their high-affinity artists or in the holdout for all of them. That choice is what makes both sides measurable at once:
- The listener side is clean by construction, since no listener is half-treated.
- The artist side becomes a within-artist comparison: for each artist, compare revenue from their exposed fans against revenue from their holdout fans, scaled by the split, and report the distribution of that per-artist lift, median first. This is why listener-level randomization matters. If you randomized pairs instead, every artist would end up with a mix of exposed and unexposed fans, so there would be no such thing as an unexposed artist, and an "exposed artists versus control artists" comparison would have nothing clean to compare.
Require the launch decision to clear a pre-registered bar on both sides (a bar both sides commit to in writing before the test starts, so neither side can redefine success once the numbers are in). If listener guardrails degrade even slightly while artist revenue rises, treat that as a stop signal, not a trade-off to argue past, because listener trust is what makes the artist-facing surface valuable in the first place.
Worked example
Illustrative sizing, not a claim about Spotify's real numbers. Say 200 million eligible listeners, and roughly 3% qualify as having strong, identifiable affinity for at least one artist: that is 6 million qualifying listeners. Suppose those listeners average 2.5 high-affinity artists each, so the surface has about 15 million listener-artist pairs to appear in. Note that the pair count is the larger number and is what drives impressions; the listener count is what drives every guardrail.
Test on 5% of qualifying listeners (300,000 listeners, therefore about 750,000 pairs) against an equal-sized listener holdout for four weeks. Suppose 2% of exposed listeners take at least one direct-support action (6,000 supporters) at an average of $4 per supporter over the four weeks, that's $24,000 of artist-attributed revenue from this 5% listener slice. That's not a number to extrapolate to "100% of the base equals 20x that," since conversion typically drops as a program moves from an early, most-receptive slice to everyone, but it's enough of a signal to judge whether this is worth a real build versus a rounding error.
Split that $24,000 by artist before drawing any conclusion. If the top 1% of artists in the test capture most of it, the mean lift per artist is meaningless and the concept has not done what it was built for. The number that matters is the median artist's within-artist lift, exposed fans versus holdout fans.
On the listener side, suppose 7-day retention and skip rate for exposed listeners land within a pre-registered 0.5 percentage point tolerance of the holdout (the "no meaningful harm" band the team agreed on before running the test), and the opt-out rate is under 3% of exposed listeners. That combination, a real and scalable-looking artist number with a healthy median and no detected listener harm against a bar set in advance, is what "working for both sides" looks like in practice, rather than a launch decided from one side's number alone.
Trade-offs and pitfalls
- The single biggest trap: measuring whichever side's metric is easiest to move (almost always the business one) and declaring victory before checking the other side's guardrail. Pre-register both thresholds before results come in, or a small listener regression gets rationalized away once the revenue number looks good.
- Concentration risk: report the median lift per artist, not just the mean, or a handful of already-successful artists can make the whole concept look like a win while it does nothing for the long tail, which undercuts the actual goal.
- Unit confusion: quoting a pair count as if it were a headcount (or the reverse) silently rescales the whole business case by the average number of high-affinity artists per listener. State which unit every number is in.
- An opt-in, well-targeted prompt still becomes annoying if shown every session; it needs a frequency cap and a genuinely respected opt-out. Note that the frequency cap has to be per listener, not per artist, or a fan of five artists gets five times the prompts.
- Affinity (how much someone listens) is not the same as willingness to pay; the pilot has to test conversion directly rather than assume it from engagement signals alone.
For a personalization feed you're about to launch, what 4 to 6 success metrics would you track that cover product, design, and engineering perspectives, not just clicks? Which of those matter most for the initial launch versus what you'd add later, and which ones are guardrails rather than goals?
Sample Answer
Direct answer
Clicks alone reward whatever gets tapped, not what actually delivers value, so a personalization
feed needs a mix across three lenses: a real value signal beyond the click, a quality and trust
signal, and a systems-health signal. I would track five: meaningful consumption rate, return rate,
content diversity, negative feedback rate, and serving latency. At initial launch, meaningful
consumption rate is the only one of these that ranking work should be trying to move; latency,
diversity and negative feedback are all guardrails from day one, with latency the strictest of them
because it is also the gate the feed has to clear before any other number can be read at all.
Return rate is a goal too, but it only becomes a reliable read once enough time has passed to
measure it.
Structured elaboration
- Meaningful consumption rate (product): the percent of surfaced items consumed past a real
threshold (watched, read, or listened past a meaningful point), not just clicked. This guards
directly against optimizing for clickbait-shaped content that gets tapped but not actually used. - Return rate (product): the share of daily active users who specifically come back to check
the feed again, a habit signal distinct from forced or incidental usage. - Content diversity or repetition rate (design): how many unique items get surfaced across
impressions, guarding against homogenization, where everyone effectively sees the same handful of
items. - Negative feedback rate (design and trust): hides, "not interested" taps, and reports per
1,000 impressions, a signal that raw clicks hide completely. - Serving latency (engineering): p95 time to render the feed. This is a functional
prerequisite, not a preference metric: if the feed loads slowly, none of the other numbers mean
anything, because users never got a fair chance to react to the content.
Launch sequencing: latency is a gate, not a growth lever, it must be acceptable before any other
metric is meaningful, so it is the primary launch read alongside meaningful consumption rate,
which tells you directionally whether the feed is working at all. Being a primary read is not the
same as being a goal, and the distinction matters because it changes what a ranking change is
allowed to trade: you gate the launch on latency and you protect it afterwards, but no ranking
change should ever be justified on the grounds that it made the feed faster, because a feed that
returns nothing returns it very quickly. Diversity and negative feedback should
be watched as guardrails from day one even though they are noisy at low volume, because the
alternative, waiting to start measuring them, means you would not notice a slow-building filter-bubble
or trust problem until it is already entrenched. Return rate needs real elapsed time and volume to
be reliable, so it naturally becomes a primary read a few weeks after launch rather than in week one.
Goals versus guardrails: meaningful consumption rate and return rate are goals, things ranking
changes should be explicitly trying to move. Negative feedback rate, diversity, and latency are
guardrails, things that must not regress while chasing the goals; diversity in particular should
never be treated as something to directly optimize upward, since chasing it as a goal can tank
short-term consumption just as easily as ignoring it can quietly produce a filter bubble.
Worked example
An illustrative week-one launch scorecard: latency p95 at 180 milliseconds (within a 200 millisecond
target, passing), meaningful consumption rate at 22% of impressions, negative feedback rate at 3 per
1,000 impressions (within a guardrail ceiling of 5 per 1,000), and 340 unique items surfaced per
1,000 impressions. These are hypothetical numbers meant to illustrate what a launch dashboard would
show, not a real measured result.
Trade-offs and pitfalls
Optimizing purely for consumption without a diversity guardrail produces filter bubbles and content
fatigue that build slowly and are invisible in week-one data, which is exactly why diversity has to
be tracked from day one even before anyone is ready to act on it. Treating a guardrail as a goal is
a subtler mistake than it sounds: deliberately trying to increase diversity as a target, rather than
protecting it as a floor, can visibly hurt short-term engagement. And engineering metrics like
latency are the ones most likely to get quietly dropped from a "success metrics" conversation
entirely, even though nothing else on the list matters if the feed does not load.
Describe the hierarchy of metrics you would set up to monitor product health (e.g., north star, leading indicators, lagging metrics). Then, for a social consumer app, propose a three-level metric tree including a north-star and two leading indicators with definitions and why they matter for problem solving.
Sample Answer
Direct answer
The hierarchy has three layers: a single north star metric at the top representing overall value
delivered, leading indicators beneath it that the team can move now and that predict where the
north star is heading, and lagging or outcome metrics that confirm value was actually captured over
a longer horizon (revenue, long-term retention). For a social consumer app, I would set the north
star as weekly connected users, users who both post or react and receive a reaction within the
week, supported by two leading indicators: new-user activation rate and content-response rate.
Structured elaboration
The point of the hierarchy is diagnostic, not decorative: when the north star moves, leading
indicators tell you where in the system the movement is coming from, before the (slower, noisier)
lagging metrics would ever reveal it.
- North star: weekly connected users, users who receive at least one reaction to something
they posted or shared during the week. This is deliberately a two-sided metric: pure posting
volume can be inflated by a few very active users, while requiring a received reaction ties the
metric to real reciprocated value on the network, not just activity. - Leading indicator 1, new-user activation rate: the percent of signups who complete a first
meaningful action (post plus receive one reaction) within 24 hours. It matters for problem
solving because it isolates whether the top of the funnel is healthy, independent of what is
happening to the existing user base. - Leading indicator 2, content-response rate: the percent of posts that get at least one
reaction or comment within 24 hours. It matters because it is a direct read on network density:
a network that has stopped responding to new content will drag the north star down within weeks,
and this metric will show the problem well before that happens.
Worked example
Say weekly connected users sit at 40,000 out of 100,000 weekly active users (40%), and it drops to
34,000 the following month. Checking the leading indicators: activation rate held steady at 55%
(new users are onboarding fine), but content-response rate on posts from days 3 through 7 of the
week fell from 62% to 48% (existing-user engagement is fading). That pattern points the
investigation at the feed ranking algorithm or a notification regression affecting existing users,
not at onboarding, and it does so weeks before a lagging metric like monthly retention would have
surfaced the same story.
Trade-offs and pitfalls
A north star built from too many combined conditions becomes unintelligible and hard for any single
team to move; keep the definition to the smallest set of conditions that still captures real,
reciprocated value. Lagging metrics like revenue or long-term retention should not drive weekly
team decisions: they move too slowly and arrive too late to be actionable at that cadence, which is
exactly why the leading-indicator layer exists. Finally, any leading indicator that a team is
directly rewarded for moving needs a guardrail alongside it (for example, reports or blocks per
session next to content-response rate), or the team will find the fastest way to inflate the number
rather than the healthiest one.
For an early-stage marketplace connecting freelance tutors with students, what would you pick as the north star metric, and why does it beat the other obvious candidates you're rejecting? Name three supporting metrics you'd track alongside it.
Sample Answer
Direct answer
I would pick weekly completed tutoring sessions as the north star metric: the count of sessions
that actually happen, not sessions booked, tutors signed up, or dollars changing hands. It beats
gross marketplace volume (GMV), total registered users, and average rating because it is the one
number that can only go up when real value gets exchanged on both sides of the marketplace at once.
Structured elaboration
A good north star metric for a two-sided marketplace has to satisfy three things: it reflects value
delivered to both sides simultaneously, it moves in the direction the business actually wants
(growth, not just activity), and it is hard to inflate by moving only one side of the market.
Here is why the obvious alternatives lose:
- GMV (gross merchandise value, total dollars transacted): GMV can rise purely from a price
increase even while fewer sessions happen and the marketplace is getting thinner. It measures
revenue, not liquidity (whether supply and demand are actually matching), and liquidity is the
thing an early marketplace lives or dies on. - Total registered tutors or students: a supply-side or demand-side vanity count. Someone can
sign up and never book or teach a single session; the number can look healthy while the
marketplace is actually dead. - Average session rating: a real quality signal, but a lagging one that says nothing about
growth, and it is trivially misleading at low volume (a 5.0 average built on three total sessions
tells you nothing). It belongs as a guardrail, not a north star.
Three supporting metrics I would track alongside weekly completed sessions:
- Fill rate: the percent of booking requests that convert into a completed session. This is
the earliest signal of marketplace liquidity and tells you whether growth is being held back by
a matching problem before the north star even moves. - Tutor utilization: average booked hours per active tutor per week. Low utilization predicts
tutor churn (a tutor earning too little leaves the platform) well before it shows up in the
completed-session count. - Repeat booking rate: the percent of students who book a second session within 30 days of
their first. This is the leading indicator that today's completed sessions will still be
happening next month, since a marketplace that only ever acquires new demand and never retains it
is not sustainable.
Worked example
Say the marketplace has 500 active tutors and 2,000 active students, and this week produces 1,200
completed sessions at an average price of $30, for $36,000 in GMV. Now suppose the team raises the
average price to $36 per session and completed sessions drop 10% to 1,080 because price-sensitive
students book less. New GMV is 1,080 x $36 = $38,880, higher than before, even though the
marketplace just got measurably less healthy: fewer students got tutored, and fill rate and repeat
booking almost certainly moved down alongside it. A team optimizing for GMV would read this as
success. A team watching completed sessions, fill rate, and repeat booking rate would catch the
regression immediately. That is the concrete case for rejecting GMV as the north star, not just an
abstract preference.
Trade-offs and pitfalls
Picking a metric that lives on only one side of the marketplace (say, tutor hours worked) biases
the team to over-invest in supply even if demand is the real constraint, or vice versa; completed
sessions forces both sides to be healthy simultaneously. The metric is also gameable if you are not
careful: a bad actor could mark sessions "completed" without real teaching happening, so it needs a
verification signal (a post-session confirmation or rating flow) and a no-show/cancellation rate
tracked as its own guardrail. Finally, this exact north star will need to evolve as the marketplace
matures. Early on, completed sessions is really a liquidity proxy; at scale, a business might add a
second layer (session quality, or GMV per active tutor) without demoting completed sessions from the
top of the tree.
Unlock Full Question Bank
Get access to all 7 Product Sense and Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.