Customer and User Obsession Questions

Direct answer

Turn the anecdote into an evidence packet that answers the questions a roadmap owner will ask: how many, who, how bad, what does it cost, and what exactly do you want them to do. A pain point is a specific problem a customer experiences while trying to get something done. Forwarding quotes shifts the work to someone else; a packet with counts, a denominator (the total you counted out of, such as 7 calls out of 42 reviewed) and a small, concrete ask makes the decision easy.

What goes in the packet

1. Problem in one sentence, stated as the customer's job and obstacle, not as a feature.
2. Count with a denominator. Distinct accounts, not calls, out of the total reviewed.
3. Segment. Which customer types (a segment is a group of similar customers, such as mid-market, meaning mid-sized companies) raise it, and what share of revenue they represent.
4. Verbatim quotes, two or three, word for word as the customer said them.
5. Behaviour data. Support tickets with a matching tag, usage events that show the struggle.
6. Cost. Time lost, workarounds, any churn (customers cancelling or leaving) or lost-deal mentions.
7. The ask. The smallest next step with an owner and a date.

Worked example (illustrative numbers)

Problem: Admins cannot see which invited users have not accepted, so they chase people by hand.
Evidence: raised in 7 of 42 customer calls this quarter. The 42 calls came from 36 distinct accounts and the 7 calls from 6 of them (one account raised it twice), so 6 of 36 accounts, about one in six.
Segment: 5 of the 6 accounts are mid-market.
Voice: "I keep a spreadsheet of who I invited."
Behaviour: 19 support tickets tagged invite status this quarter.
Cost: about 20 minutes a week per admin (self-reported).
Ask: a two-week discovery spike, one Product Manager and one engineer, with a go or no-go decision at the end. Owner: the Product Manager for the admin and invitations area. Dates: spike starts the first Monday of next month, go or no-go review on the Friday two weeks later, with the go criteria agreed before it starts.

A discovery spike is a short, time-boxed investigation (here two weeks) that learns enough to decide whether to build, without building the feature. A go or no-go decision is a yes or no on whether to proceed; the criteria for that yes or no are agreed before the spike starts.

6 of 36 accounts is 16.7 percent, so "about one in six" is accurate (7 of 42 calls happens to give the same 16.7 percent here, but the account count is the figure to quote). Counting distinct accounts against distinct accounts keeps one chatty customer from inflating the number.

Making it land

• Take it to the owner of that product area, framed against current priorities: what it could displace and where it might fit.
• Tag every call consistently from now on, in a shared tracker, so counts accumulate instead of resetting.
• Say what you do not know, such as whether the problem is as common among customers who do not take calls.

Trade-offs and pitfalls

• Customers who book calls are not a random sample of your base (sampling bias: the people you hear from differ from the people you do not). Flag it and pair call counts with behaviour data.
• Do not escalate with urgency theatre (acting as if everything is on fire to get attention). A modest, well-evidenced ask is more likely to be funded than a loud one.
• A small ask (a spike) lets the roadmap owner say yes without reshuffling the whole plan.

Direct answer

I would ask rarely, at the right moment, of a sample, and give people an always-available way to volunteer feedback, then compare the two so loud, angry voices do not stand for everyone. What comes back is tagged, linked to usage context, counted by theme, and answered, so people see that it matters.

Design: when to ask

• At a natural completion point, such as right after a user finishes a task, not mid-task. Never interrupt checkout, error recovery or a first session.
• Eligibility rules (the conditions a user must meet before being shown the prompt): only users who have done the relevant thing a few times, and suppress (hide the prompt from) anyone who dismissed a prompt recently.
• Frequency cap (a limit on how often the same user is asked): one prompt per user per fixed window (for example 60 days, a design choice to tune).
• Sample, do not blast: show the prompt to a random slice of eligible users.

Illustrative sizing: with 20,000 eligible users a week and a 10% sample, 2,000 see the prompt. At an assumed 10% response rate (responses divided by users shown the prompt), that is about 200 answers a week.

Design: how to ask

• One quick question (a rating or thumbs up or down) plus an optional free-text box.
• Dismissible in one tap, and the dismissal is remembered.
• Capture context automatically (page, app version, plan, session ID) with clear consent, so users do not have to describe where they were.

Example (illustrative): right after a user exports a report successfully, a small card appears at the bottom of the screen, not a blocking pop-up. It reads "How was exporting that report?" with thumbs up and thumbs down. A tap opens an optional box, "Anything we should know?", and a "Not now" link closes the card for 60 days. The question is about the task just finished, not about the product in general.

Avoiding the angriest-only problem

A permanent "send feedback" button draws people with strong feelings, usually bad ones. A sampled prompt at a moment of success reaches satisfied and neutral users too. Keep both channels, label them in reports, and compare responders with the user base (plan, tenure, usage) to see who is missing.

Variants

For AI assistants, route by severity: harmful or high-stakes errors escalate immediately, ordinary quality complaints go into the weekly theme review.

What to do with the responses

1. Tag by theme, attach usage context, and count distinct users per theme.
2. Weekly review: urgent issues to engineering, repeated themes to the backlog, how-to questions to docs.
3. Reply to people who left contact details, and tell them when it ships.
4. Track the health of the channel itself: response rate by segment and share of respondents who are negative versus the base's behaviour.

Pitfalls

Over-asking trains users to dismiss the prompt. Treating the average rating as a verdict hides who answered. Collecting feedback and never replying teaches users to stop giving it.

Direct answer

"Search is slow" is a symptom described in the customer's words. To make it actionable I would translate it into a measurable problem statement: who is affected, what they experience, how we measure it, and what good looks like. Then I would test that the measurement explains the complaints, rather than assuming the first reading is right.

Step 1: Quantify the complaints

Tag a sample of comments and tickets by what "slow" means:

• Waiting a long time for results to appear (latency).
• Results appear but are the wrong ones, so the user searches again and again (relevance, which feels slow).
• The page gives no sign it is working, so a short wait feels long (feedback).

Illustrative tagging of 60 comments: 31 describe waiting (a few of these also say the page gave no sign of progress, the third meaning, so a comment can carry more than one tag), 14 say they cannot find things, 15 are unrelated. Then count distinct customers, plan tiers and dates, and note whether they began after a release.

Step 2: Join the complaints to telemetry (the usage and timing data the product records automatically)

For the customers who complained, look at the search response time (use the 95th percentile, the time that 95% of searches beat, as well as the median, the typical middle search), searches per session, and the reformulation rate, the share of searches followed within seconds by a rewritten search. Compare with customers who did not complain. If complainers are mostly accounts with very large data sets, that is a precise lead for engineering.

Step 3: Write the engineering-ready ticket

Problem: Accounts with more than 100,000 records wait too long for search results.
Evidence: 31 of 60 sampled comments mention waiting. For those accounts P95 search
          time is 6.8 s against 1.9 s for other accounts (illustrative numbers).
Metric:   P95 search response time per account tier, measured at the server and
          at the browser when results render. Baseline today: 6.8 s.
Target:   P95 of 3 s or less for the tier (illustrative; the real value is agreed
          with engineering from the measured baseline and user tolerance).
Hypotheses (candidate causes to test): index size; a recent release; query type;
          missing loading state.
Done when: target met for the tier AND complaint volume and reformulation rate fall.

Step 4: How I know it is the real problem

• Reproduce it on a complaining customer's account, or watch a screen recording of their session.
• Check timing: did complaints start when the latency measure worsened? If complaints came first, suspect relevance or perception.
• Check the dose-response (more of the suspected cause should go with more of the effect): do the accounts with the worst measured latency complain most? Illustratively, sort 100 accounts by P95 and cut them into four equal groups. If complaints per 100 users run 1, 2, 4 and 9 from the fastest group to the slowest, the pattern supports latency as the cause. If all four groups sit near 3, latency is not what customers are reacting to.
• After the fix, confirm the complaint rate drops and a behavioural metric (reformulations, abandoned searches) improves. If latency improves but complaints do not, we fixed a different problem.

Trade-offs and pitfalls

• Setting a performance target before measuring. Targets come from baseline and user tolerance, not intuition.
• Handing engineering "customers hate search" with no evidence. They cannot prioritise or verify it.
• Treating ticket volume as a measure of breadth: one customer can file 30 tickets.

Direct answer

I would default to the customer-visible onboarding change, unless the QA bottleneck is slowing everything the team ships. The decision turns on one comparison: how much customer value does each option create, on a common basis, for the same engineering cost? Internal tools are not worse in principle (faster QA means faster delivery of everything), but their benefit is indirect, so I would need evidence that the bottleneck is real.

Put both options on the same scale

• Customer-visible change: what share of new users fail to finish onboarding, and what would a fix plausibly move? Measure activation (reaching the first meaningful success, such as the first completed task).
• Internal tool: hours of QA time saved, delay between code complete and release, and bugs reaching customers.
• Convert both to "what would customers feel, and when?" For the tool, that is releases that arrive sooner or with fewer defects.

Evidence I would want

1. Onboarding funnel (the sequence of onboarding steps, with the share of users who drop off at each): where users leave, and what they say in support or session recordings (videos of real users' screens).
2. Cost of the fix: engineering weeks for each option, and how confident that estimate is.
3. QA data: how much time per release goes to the manual work, and whether QA is on the critical path of releases (the slowest chain of steps, so that if QA is slow, every release waits and can be held back until its checks pass).
4. A cheap test of the onboarding fix (a prototype shown to a few users) before committing.

Worked example (illustrative numbers)

Assume both options cost the same, about 6 engineer-weeks.

Onboarding: 10,000 signups a month, 40% finish onboarding, and the fix is hoped to raise that to 44%. That target is an assumption to test, but if it holds, 400 more activated users a month (4 points of 10,000).

QA tool: 3 QA engineers lose 6 hours each a week to the manual steps, so 18 hours a week, about 78 hours a month (18 x 4.33), roughly two working weeks of QA time. Customers feel it only if those hours would otherwise gate a release.

Putting both in one unit: dollars over the first 12 months. Illustrative assumptions: 25% of activated users become paying customers at $20 a month, and QA time is worth $50 an hour.

Onboarding: 400 activated x 25% = 100 new payers a month = $2,000 of new monthly revenue each month
  12 monthly cohorts, the first paying 12 months, the last 1 month: $2,000 x (12+11+...+1) = $2,000 x 78 = $156,000
  (ignores churn and build time)
QA tool:    78 hours x $50 = $3,900 a month x 12 = $46,800 (the ceiling, since it is only worth that if the freed time is put to use)

So onboarding is about three times larger, and the verdict holds unless fewer than about 7.5% of activated users pay ($46,800 / ($400 x $20 x 78) = 0.075). That payer rate is the number I would test first. One basis caveat: the onboarding figure is gross revenue while the QA figure is the value of saved labor, so they are not strictly the same kind of money. Taking an assumed 80% gross margin on the revenue, the break-even payer rate rises from 7.5% to about 9.4% ($46,800 / ($400 x $20 x 78 x 0.8) = 0.094), so the ranking survives but with a thinner margin than the headline three-times suggests. If the data showed releases waiting two days on manual QA every week, the tool would speed up every future customer change and I would flip.

Trade-offs and pitfalls

• The visible change risks being a guess about what users want. Mitigate with a quick test.
• Internal tools get chronically underfunded, so a rule such as a small standing share of capacity for team productivity prevents a permanent "later."
• Do not decide on whoever argues loudest.

Direct answer

Treat the two requests as evidence about different jobs to be done (the outcomes each group is trying to achieve) for different segments (a segment is a group of customers with similar needs and buying behavior), not as two opinions to pick between. My recommendation: keep the simple experience as the default for everyone, and give enterprise depth through layers (admin settings, an integration API, features switched on per account) instead of adding steps to the main flow. Split into separate products only if the evidence shows the two jobs cannot share a core.

Framework: four questions in order

1. Who is asking, and how much is behind it? Enterprise: few accounts, high annual recurring revenue (ARR) each, loud and specific requests. Consumer: many users, small revenue each, and the request ("simple, fast") shows up in behavior (drop-off, time to first success, meaning how long a new user takes to reach a first useful result) more than in tickets.
2. What job sits under each request? "Deep customization" usually means "make this fit our approval process and our other systems." "Simplicity and speed" means "reach my first result without learning anything." Five enterprise accounts asking for five different custom fields is one need (configurable fields) in five costumes.
3. Is the conflict real? It is real only if serving the enterprise need adds steps or options to the consumer path. If it can sit behind an admin setting, there is no conflict to resolve.
4. Score what survives on value (revenue at stake plus users affected), cost (build plus ongoing maintenance), strategic fit, and reversibility. Prefer the option that is cheap to undo.

Who is involved and what each contributes

• Sales: the real deal and renewal evidence, separating "lost the deal over this" from "asked for it on a call."
• Design: prototype progressive disclosure (advanced options appear only when the user asks for them) and measure the complexity cost on the consumer path.
• Engineering: the maintenance cost of each option, and which pieces are reusable platform versus one-off code.
• I make the call and write down why. Each group commits to its part, so nobody learns the decision from a release note.

Worked example (illustrative numbers)

A team task app has a self-serve segment with tens of thousands of small teams and 12 contracted enterprise accounts. Enterprise asks: custom approval steps, single sign-on options (logging in with the company's own identity system), a sync to their ticketing tool. Self-serve data: many new teams stall before creating their first task.

Proposal for the quarter:

• Segmentation: self-serve versus contracted enterprise, tracked separately in every metric.
• Ship a webhook (an automatic message our product sends to another system when something happens) and a REST integration surface (a standard web interface other software calls to read and change data) plus an admin console behind an entitlement (a per-account record of which capabilities that customer has paid for) feature flag (a switch that turns a capability on for chosen accounts without a separate release). Enterprise needs land as configuration or API use, never per-customer code forks.
• Leave the self-serve onboarding path untouched. Guardrail (a metric that must not get worse): median time to first completed task in the flag-off cohort (the users who do not have the new capabilities switched on) must not get worse; if it does, the release is rolled back.
• Fund a self-serve onboarding fix in the same quarter so the consumer segment is not the one that waits.

Scoring step, shown with illustrative scores from 1 (poor) to 5 (good), where for cost 5 means cheapest and for reversibility 5 means easiest to undo, equal weights:

C wins because it keeps most of A's value at lower cost and stays easy to undo. If sales shows that three renewals depend on a single enterprise ask, value for A rises and the totals are worth re-scoring.

Trade-offs and pitfalls

• Flags accumulate debt. Give each flag an owner and a review date, or the product becomes the complicated thing consumers feared.
• Counting enterprise requests by volume lets the loudest account set direction; count by underlying job and by ARR affected.
• Layering is not free: the API becomes a support promise. I would flip to option D if the enterprise code path repeatedly delays consumer releases, or to A if enterprise becomes the clear majority of revenue.