Product Sense and Design Questions

Bottom line

Anchor the concept on a listener behavior that already exists (personal fandom for a handful of artists) rather than a promotional push, because a concept that channels money to artists as a side effect of something listeners already want to do is far less likely to feel like an ad, and both sides need pre-committed success thresholds, not one side's number used to wave away the other.

How to decide

1. Frame it with Jobs-to-be-Done (JTBD, a way of describing what a user is really trying to accomplish, independent of any specific feature): artists want a way to convert the fans who already love them into recurring support (tips, merch, tickets), not just streaming royalties, which most artists experience as too thin to matter individually. Listeners want to discover and feel closer to artists they already care about. The right concept sits at that intersection.

2. Concept: an opt-in "Direct Support" surface that appears only for a listener's small set of high-affinity artists (based on existing signals like play count concentration and saves, not a blanket promotion shown to everyone), offering a low-friction way to support that specific artist directly: a small tip, an early-access bundle, or a merch link. It never autoplays, never interrupts a session, and is easy to permanently dismiss.

3. Name the business assumptions this concept rests on, and treat each as something to test, not something to assume:

   • A meaningful share of listeners already have strong, identifiable affinity for a small number of artists.
   • A contextual, low-friction ask converts at a nonzero rate even though a blanket ad would not.
   • The ask doesn't need to be frequent to matter in aggregate, given catalog scale.
   • It doesn't simply pull revenue away from existing subscription revenue.

4. Fix the unit of analysis before writing down a single number, because this concept has two of them and they are not interchangeable. A qualifying listener is one person with at least one high-affinity artist. A listener-artist pair is one place the surface can appear, and a listener with three high-affinity artists produces three pairs. Reach and impressions are counted in pairs; retention, opt-out, and every listener-side guardrail are counted per listener. Sliding between the two is the easiest way to be off by the average number of high-affinity artists per fan without noticing.

5. Metrics have to cover both sides, since the prompt is explicit that one-sided success doesn't count:

   • Artist side: incremental revenue per artist, and importantly the median lift, not just the average, since a handful of superstar artists can make an average look great while most artists in the surface see almost nothing.
   • Listener side: retention and session frequency for exposed listeners versus a holdout of comparable listeners not shown the surface, plus a direct opt-out or "hide this" rate as a guardrail.

6. The actual test: randomize at the listener level, not at the pair level, and not as a simple before-and-after comparison. Each qualifying listener is either in the surface for all of their high-affinity artists or in the holdout for all of them. That choice is what makes both sides measurable at once:

   • The listener side is clean by construction, since no listener is half-treated.
   • The artist side becomes a within-artist comparison: for each artist, compare revenue from their exposed fans against revenue from their holdout fans, scaled by the split, and report the distribution of that per-artist lift, median first. This is why listener-level randomization matters. If you randomized pairs instead, every artist would end up with a mix of exposed and unexposed fans, so there would be no such thing as an unexposed artist, and an "exposed artists versus control artists" comparison would have nothing clean to compare.
     Require the launch decision to clear a pre-registered bar on both sides (a bar both sides commit to in writing before the test starts, so neither side can redefine success once the numbers are in). If listener guardrails degrade even slightly while artist revenue rises, treat that as a stop signal, not a trade-off to argue past, because listener trust is what makes the artist-facing surface valuable in the first place.

Worked example

Illustrative sizing, not a claim about Spotify's real numbers. Say 200 million eligible listeners, and roughly 3% qualify as having strong, identifiable affinity for at least one artist: that is 6 million qualifying listeners. Suppose those listeners average 2.5 high-affinity artists each, so the surface has about 15 million listener-artist pairs to appear in. Note that the pair count is the larger number and is what drives impressions; the listener count is what drives every guardrail.

Test on 5% of qualifying listeners (300,000 listeners, therefore about 750,000 pairs) against an equal-sized listener holdout for four weeks. Suppose 2% of exposed listeners take at least one direct-support action (6,000 supporters) at an average of $4 per supporter over the four weeks, that's $24,000 of artist-attributed revenue from this 5% listener slice. That's not a number to extrapolate to "100% of the base equals 20x that," since conversion typically drops as a program moves from an early, most-receptive slice to everyone, but it's enough of a signal to judge whether this is worth a real build versus a rounding error.

Split that $24,000 by artist before drawing any conclusion. If the top 1% of artists in the test capture most of it, the mean lift per artist is meaningless and the concept has not done what it was built for. The number that matters is the median artist's within-artist lift, exposed fans versus holdout fans.

On the listener side, suppose 7-day retention and skip rate for exposed listeners land within a pre-registered 0.5 percentage point tolerance of the holdout (the "no meaningful harm" band the team agreed on before running the test), and the opt-out rate is under 3% of exposed listeners. That combination, a real and scalable-looking artist number with a healthy median and no detected listener harm against a bar set in advance, is what "working for both sides" looks like in practice, rather than a launch decided from one side's number alone.

Trade-offs and pitfalls

• The single biggest trap: measuring whichever side's metric is easiest to move (almost always the business one) and declaring victory before checking the other side's guardrail. Pre-register both thresholds before results come in, or a small listener regression gets rationalized away once the revenue number looks good.
• Concentration risk: report the median lift per artist, not just the mean, or a handful of already-successful artists can make the whole concept look like a win while it does nothing for the long tail, which undercuts the actual goal.
• Unit confusion: quoting a pair count as if it were a headcount (or the reverse) silently rescales the whole business case by the average number of high-affinity artists per listener. State which unit every number is in.
• An opt-in, well-targeted prompt still becomes annoying if shown every session; it needs a frequency cap and a genuinely respected opt-out. Note that the frequency cap has to be per listener, not per artist, or a fan of five artists gets five times the prompts.
• Affinity (how much someone listens) is not the same as willingness to pay; the pilot has to test conversion directly rather than assume it from engagement signals alone.

Direct answer

I would pick weekly completed tutoring sessions as the north star metric: the count of sessions
that actually happen, not sessions booked, tutors signed up, or dollars changing hands. It beats
gross marketplace volume (GMV), total registered users, and average rating because it is the one
number that can only go up when real value gets exchanged on both sides of the marketplace at once.

Structured elaboration

A good north star metric for a two-sided marketplace has to satisfy three things: it reflects value
delivered to both sides simultaneously, it moves in the direction the business actually wants
(growth, not just activity), and it is hard to inflate by moving only one side of the market.

Here is why the obvious alternatives lose:

• GMV (gross merchandise value, total dollars transacted): GMV can rise purely from a price
  increase even while fewer sessions happen and the marketplace is getting thinner. It measures
  revenue, not liquidity (whether supply and demand are actually matching), and liquidity is the
  thing an early marketplace lives or dies on.
• Total registered tutors or students: a supply-side or demand-side vanity count. Someone can
  sign up and never book or teach a single session; the number can look healthy while the
  marketplace is actually dead.
• Average session rating: a real quality signal, but a lagging one that says nothing about
  growth, and it is trivially misleading at low volume (a 5.0 average built on three total sessions
  tells you nothing). It belongs as a guardrail, not a north star.

Three supporting metrics I would track alongside weekly completed sessions:

1. Fill rate: the percent of booking requests that convert into a completed session. This is
   the earliest signal of marketplace liquidity and tells you whether growth is being held back by
   a matching problem before the north star even moves.
2. Tutor utilization: average booked hours per active tutor per week. Low utilization predicts
   tutor churn (a tutor earning too little leaves the platform) well before it shows up in the
   completed-session count.
3. Repeat booking rate: the percent of students who book a second session within 30 days of
   their first. This is the leading indicator that today's completed sessions will still be
   happening next month, since a marketplace that only ever acquires new demand and never retains it
   is not sustainable.

Worked example

Say the marketplace has 500 active tutors and 2,000 active students, and this week produces 1,200
completed sessions at an average price of $30, for $36,000 in GMV. Now suppose the team raises the
average price to $36 per session and completed sessions drop 10% to 1,080 because price-sensitive
students book less. New GMV is 1,080 x $36 = $38,880, higher than before, even though the
marketplace just got measurably less healthy: fewer students got tutored, and fill rate and repeat
booking almost certainly moved down alongside it. A team optimizing for GMV would read this as
success. A team watching completed sessions, fill rate, and repeat booking rate would catch the
regression immediately. That is the concrete case for rejecting GMV as the north star, not just an
abstract preference.

Trade-offs and pitfalls

Picking a metric that lives on only one side of the marketplace (say, tutor hours worked) biases
the team to over-invest in supply even if demand is the real constraint, or vice versa; completed
sessions forces both sides to be healthy simultaneously. The metric is also gameable if you are not
careful: a bad actor could mark sessions "completed" without real teaching happening, so it needs a
verification signal (a post-session confirmation or rating flow) and a no-show/cancellation rate
tracked as its own guardrail. Finally, this exact north star will need to evolve as the marketplace
matures. Early on, completed sessions is really a liquidity proxy; at scale, a business might add a
second layer (session quality, or GMV per active tutor) without demoting completed sessions from the
top of the tree.

Bottom line

Treat pricing as a series of small, real tests rather than a single guess, and instrument cannibalization from day one, meaning you directly measure how many buyers would have paid more anyway, rather than assuming a high conversion number is automatically good news.

How to decide

1. Scope the question precisely: price the value of the one specific enhanced feature being unlocked, not the whole product. Bundling more into the test than that one capability makes it unclear what buyers are actually paying for.

2. Get a cheap read on willingness to pay before setting any real price: a short survey (asking, for instance, at what price the feature would feel too expensive versus too cheap to trust), plus a proxy signal you likely already have, current usage intensity of the free version of that capability. Someone who already uses it heavily is a stronger candidate to pay than someone who's never touched it.

3. Run a live pricing test: randomize a few price points across new signup cohorts, never silently change price for existing paying customers, and measure conversion at each price.

4. Measure cannibalization directly rather than assuming it away: for buyers at the new tier, estimate how many were already on a path to becoming full-tier customers and instead settled for the cheaper option (a "downgrade capture" effect), versus genuinely new revenue from people who wouldn't have paid at all.

5. Decide on net incremental revenue, not raw conversion rate:

$\text{Net incremental revenue} = \text{new tier revenue} - \text{cannibalized revenue}$

A price that produces the highest conversion rate can still be the wrong choice if a large share of those buyers would have paid more anyway.

6. Know in advance how much damage cannibalization can actually do at each price, because that is a bounded quantity you can compute before any data arrives. The worst case is that every new-tier buyer was a would-be full-tier customer, so the ceiling on cannibalized revenue at a given price is (buyers at that price) x (the historical upgrade rate for that profile) x (full-tier price). Compare that ceiling to the arm's own revenue. If the ceiling is below the revenue, the arm cannot go net negative no matter what the data says, and the interesting question is not "is it positive" but "does it still beat the other price points."

Worked example

Illustrative numbers for a fictional note-taking app, "TidyNotes." Today: 500,000 free users, 20,000 already pay $8/month for the existing "Pro" tier (4% of the free base converted, $160,000 monthly recurring revenue). The team considers unbundling an advanced export feature, currently Pro-only, into a standalone add-on for free users.

Test: randomize three price points ($2, $3, $5 per month) across three equal new-signup cohorts of 10,000 each, leaving existing Pro pricing untouched. After one billing cycle:

• $2: 6% buy (600 buyers) = $1,200 monthly revenue from that cohort.
• $3: 4.5% buy (450 buyers) = $1,350 monthly revenue from that cohort.
• $5: 2% buy (200 buyers) = $1,000 monthly revenue from that cohort.

By raw revenue, $3 looks best. Now check cannibalization: of the 450 buyers at $3, suppose usage data shows 90 of them (20%) were already heavy users of the free export feature who, based on the app's historical conversion pattern for that usage profile, would have upgraded to full $8 Pro at roughly a 25% rate within 90 days even without the cheaper add-on. Expected Pro revenue given up: 90 x 0.25 x $8 = $180/month. Net incremental revenue at $3 is $1,350 minus $180, or $1,170.

Before treating that as settled, compute the ceiling. Even if all 450 buyers at $3 were near-converters, cannibalized revenue would be 450 x 0.25 x $8 = $900, still less than the arm's $1,350. So at these numbers the $3 add-on cannot be net negative; you would need the historical upgrade rate to be above 37.5% (not 25%) before 100% near-converters could wipe out the arm. Knowing that ceiling stops the team from arguing about a risk the arithmetic has already ruled out.

What cannibalization can do is reorder the price points, and that is the finding worth chasing. Cannibalized revenue scales with the number of buyers, so the cheapest price, which wins the most buyers, is also the most exposed. Rerun all three arms assuming a much worse 80% near-converter share instead of 20%:

At 20% near-converters, $3 wins ($1,170). At 80%, the ranking flips and $5 wins ($680 versus $630), while $2 collapses from a headline-best 6% conversion rate to the worst option on the board. Nothing about the headline conversion numbers changed; only the estimate of who would have paid more anyway did. That is the entire argument for measuring the near-converter share directly rather than inferring quality from conversion rate, and it is also the reason to compute this sensitivity band before launch rather than after.

Trade-offs and pitfalls

• Testing only on new signups avoids a self-inflicted backlash from changing prices on existing customers, but it also means the test doesn't directly observe whether existing Pro subscribers would downgrade once a cheaper option exists; that risk needs its own careful, opt-in test if it's a real concern, and it is not bounded by the ceiling calculation above, since the existing 20,000 Pro subscribers are a much larger pool than any new-signup cohort.
• The price that maximizes short-term conversion isn't automatically the price that maximizes long-term revenue if it trains a chunk of your best future full-tier customers to permanently settle for less. The "would have converted anyway" estimate is inherently uncertain, so report it as a range and show the decision across that whole range, as in the table above, rather than picking one point estimate and defending it. Revisit with real data once the cohort ages past 90 days.
• The most common wrong turn is reporting conversion rate as the win metric and stopping there, without ever estimating what it cost the existing or future higher tier.
• Testing more than one bundled capability at once makes it hard to tell what price signal is actually about; keep the test scoped to one clearly-defined capability where possible.

Direct answer

A strong candidate resists jumping straight to "add a toolbar button" or "don't," and instead treats this as a discoverability problem with more than two solutions. The real question is how to help more users benefit from the shortcut's speed without paying the UI-clutter cost of a permanent, always-visible button, and there are usually cheaper, more targeted options in between those extremes.

How to evaluate it

1. Quantify current impact first: instrument the shortcut's usage, how many users rely on it, how much time or how many steps it actually saves, and whether its users skew toward an identifiable segment doing a specific repeated task.
2. Talk to a handful of people already using it: how did they discover it, and what would have helped them find it sooner? This tells you whether the real problem is that nobody knows it exists, or that the workflow itself is clunky and the shortcut is a workaround.
3. List alternatives to a permanent toolbar button, in roughly increasing order of visual cost: a contextual hint that surfaces the shortcut only when the relevant action is available; a searchable command palette, a lightweight, keyboard-driven menu of actions that scales to many features without adding a button for each one; a user-customizable toolbar so power users can add it themselves without changing the default for everyone; and, last, a permanent toolbar button, reserved for features valuable and broadly used enough to earn permanent screen space.
4. This is progressive disclosure in action, a design principle where advanced or less-common options are revealed only when relevant, so the default experience stays simple for most users while power users can still reach the depth they need.

Measuring the effect

Run a test with at least three groups: no change, a low-cost option such as a hint or command palette, and the toolbar button if the team still wants to test it directly. Primary metrics: completion time or step count for the target workflow, and adoption rate of whichever surfacing method is tested. Also watch for a regression in the other direction: if a permanent button measurably slows down or confuses users who never use the shortcut, more mis-clicks, longer time to find other toolbar items, that's a real cost, not a hypothetical designer worry.

Decide before you look at the data which of two numbers you are judging on, because for an opt-in surface they behave very differently:

• The arm-level (intent-to-treat) number, everyone assigned to the arm whether or not they ever touched the new surface. This is the honest estimate of what shipping the change does to the product.
• The adopter number, only the people who actually used the new surface. This is the size of the benefit when it lands, but it is not causal on its own, because adopters self-select and were probably faster or more engaged to begin with.

Worked example

Suppose the team compares no change against a command palette that includes the shortcut as a discoverable action, tested on about 2,000 users per arm. After two weeks, roughly 15% of the palette arm (about 300 of 2,000 users) has adopted it, and it is used almost entirely by people who were not using the raw keyboard shortcut before.

Among those adopters, median task time on the target workflow falls from 45 seconds to about 24 seconds, a 47% reduction, which is the real size of the effect when someone actually finds the palette. The other 85% are unchanged, with their median still at 45 seconds.

Now roll that up to the arm, because this is the number a launch decision has to be made on: 0.85 x 45 + 0.15 x 24 = 41.9 seconds, an average improvement of about 3.1 seconds, or 7%, per assigned user. The arm's median barely moves at all and stays at roughly 45 seconds, because 85% of the arm is untouched and the 50th percentile of the arm still sits inside that untouched group. A median is the wrong summary statistic for a change only a minority opts into; report the adoption rate and the adopter effect separately, or the headline reads as "no effect" when a real 47% win exists inside a 15% slice.

That pattern argues for the low-cost option over a permanent toolbar button: it captures a large per-adopter benefit at almost no ongoing UI cost, and the way to grow the arm-level number is to raise the 15% adoption rate, not to spend permanent screen space.

Trade-offs and pitfalls

Jumping straight to exposing it as a button without measuring who benefits risks cluttering the interface for a workflow only a small minority uses. Jumping straight to keeping it hidden to protect simplicity, without testing anything, risks leaving real, measurable productivity on the table for engaged users. Reporting only the adopter number is the most common way this kind of test gets oversold, since a 47% improvement among self-selected adopters is not a 47% improvement to the product. And a shortcut used heavily by a narrow, high-value segment, say your most active daily users, might deserve visibility even at a modest cost to everyone else; the decision should weigh who the users are, not just how many there are.

Direct answer

A strong candidate defines the MVP, the minimum viable product, meaning the smallest version of the product that lets you learn something real from actual users, as the smallest set of capabilities that lets a third-party developer sign up, find what they need, and successfully make one real call to the API. That moment of a first successful call is where you learn whether the integration story actually works, so everything not required to reach it gets cut for version one.

What's in, what's out

In (core): self-serve sign-up and API key issuance, a runnable quickstart with a copy-pasteable example (not just prose), the essential reference docs covering endpoints, authentication, rate limits, and error codes, and a way for a developer to tell you when they're stuck, even if that's just an email address someone actually reads.

Out, intentionally, for version one: multiple language SDKs, a partner marketplace or billing and monetization, fine-grained role-based access control, and a polished analytics dashboard for partners. These all matter eventually, but none of them changes whether the core integration works.

Acceptance criteria for launch

Write these as a number and a way to check it, not an adjective. Concretely, for this launch:

• At least 80% of the developers in the pilot go from sign-up to one successful API call in under 30 minutes with no support contact, measured from instrumented timestamps rather than self-report. Thirty minutes is one uninterrupted sitting: past that a developer context-switches away and comes back a day later, if at all. 80% is where the remaining failures are plausibly individual rather than systemic. Both numbers are a pre-pilot hypothesis and both get re-cut after the first pilot, but they get written down first, because a criterion with no number in it cannot fail a launch review, which is the only thing a launch criterion is for.
• The reference docs cover 100% of the endpoints, auth steps, and error codes touched by the one supported example flow, not the entire API surface, verified by walking that flow against the published docs alone with no internal knowledge and no Slack access.
• A feedback channel exists, is linked from the quickstart itself, has a named owner, and has a stated response target of one business day, so "someone is watching it" is a checkable claim instead of an intention.

Worked example

Validate the MVP by recruiting a small group of real target developers, partners or engaged community members rather than internal staff, for a short pilot (roughly two weeks is enough to see real usage patterns). Instrument every attempt (time-to-first-call, drop-off point, errors hit) and pair each metric with a short debrief question like "what almost made you give up?" Suppose 15 external developers try it: 11 of 15 make a successful call within 20 minutes, and the other 4 all stall at the same step, an unclear error message when their API key hasn't finished propagating yet. Note that 11 of 15 is 73%, under the 80% launch bar above, so this pilot does not clear it. That one repeated failure point is worth more than the aggregate "11 of 15 succeeded" number, because it tells you exactly what to fix before a wider launch, while the aggregate alone would just say things are mostly fine.

Trade-offs and pitfalls

Cutting too aggressively (no working example, only prose docs) means nobody reaches the moment of truth at all, and you learn nothing. Over-scoping the MVP by adding an SDK "because partners will ask" delays your first real signal and spends effort on something you haven't validated is needed yet. And acceptance criteria written as vague aspirations like "good developer experience" can't fail a launch review; write them as a number and a method for checking it, even if that number is a rough hypothesis at first.