Customer and User Obsession Questions

Direct answer

I would ask rarely, at the right moment, of a sample, and give people an always-available way to volunteer feedback, then compare the two so loud, angry voices do not stand for everyone. What comes back is tagged, linked to usage context, counted by theme, and answered, so people see that it matters.

Design: when to ask

• At a natural completion point, such as right after a user finishes a task, not mid-task. Never interrupt checkout, error recovery or a first session.
• Eligibility rules (the conditions a user must meet before being shown the prompt): only users who have done the relevant thing a few times, and suppress (hide the prompt from) anyone who dismissed a prompt recently.
• Frequency cap (a limit on how often the same user is asked): one prompt per user per fixed window (for example 60 days, a design choice to tune).
• Sample, do not blast: show the prompt to a random slice of eligible users.

Illustrative sizing: with 20,000 eligible users a week and a 10% sample, 2,000 see the prompt. At an assumed 10% response rate (responses divided by users shown the prompt), that is about 200 answers a week.

Design: how to ask

• One quick question (a rating or thumbs up or down) plus an optional free-text box.
• Dismissible in one tap, and the dismissal is remembered.
• Capture context automatically (page, app version, plan, session ID) with clear consent, so users do not have to describe where they were.

Example (illustrative): right after a user exports a report successfully, a small card appears at the bottom of the screen, not a blocking pop-up. It reads "How was exporting that report?" with thumbs up and thumbs down. A tap opens an optional box, "Anything we should know?", and a "Not now" link closes the card for 60 days. The question is about the task just finished, not about the product in general.

Avoiding the angriest-only problem

A permanent "send feedback" button draws people with strong feelings, usually bad ones. A sampled prompt at a moment of success reaches satisfied and neutral users too. Keep both channels, label them in reports, and compare responders with the user base (plan, tenure, usage) to see who is missing.

Variants

For AI assistants, route by severity: harmful or high-stakes errors escalate immediately, ordinary quality complaints go into the weekly theme review.

What to do with the responses

1. Tag by theme, attach usage context, and count distinct users per theme.
2. Weekly review: urgent issues to engineering, repeated themes to the backlog, how-to questions to docs.
3. Reply to people who left contact details, and tell them when it ships.
4. Track the health of the channel itself: response rate by segment and share of respondents who are negative versus the base's behaviour.

Pitfalls

Over-asking trains users to dismiss the prompt. Treating the average rating as a verdict hides who answered. Collecting feedback and never replying teaches users to stop giving it.

Direct answer

Treat the conflict as a hidden disagreement about outcomes, not features. In business-to-business (B2B) products the economic buyer (the person who controls the budget) usually asks for a requirement that protects some goal such as control, compliance or visibility, while end users need to get daily work done quickly. A product users avoid does not get renewed, and a product the buyer cannot justify does not get bought. So I find the buyer's real goal, measure the actual cost to users, and look for a design that delivers the goal with the least user burden. Only if none exists do I escalate it as an explicit trade-off.

Steps

1. Name the conflict in one sentence with both sides. "The buyer requires X; users report Y."
2. Ask the buyer why. "What would you be able to show or prevent if this were in place?" Requirements are often proxies: a stand-in request for the real goal behind it. "Mandatory fields" may really mean "an audit trail".
3. Quantify user cost. Extra time per task, share of users affected, whether they already avoid the step, adoption trend. For example: 8 extra seconds per task, 15 tasks per shift is 2 minutes per user per shift; 120 of 150 users affected; 30% of entries already skipped or filled with the default.
4. Lay out options and who pays for each:

5. Decide with both in the room. Show the buyer the usage evidence; show users the reason. A hard requirement (legal, regulatory, written into the contract) outranks preference; a preference loses to demonstrated workflow harm.

Worked example (illustrative)

An operations director buying a warehouse system requires pickers to enter a reason code every time an item is short, to reduce shrinkage reports (shrinkage is stock lost or unaccounted for, such as theft or miscounts). A reason code is a short label from a fixed list, such as "damaged" or "miscounted". Pickers hit shorts many times a shift, the extra typing slows them, and they start choosing the first code in the list just to move on, so the data becomes unreliable. Resolution: preselect the most likely reason from scan context (the scan trail: the record of what was scanned, where and when), ask only when the situation is unusual, and give the director a weekly report built from the scan trail. The buyer gets trustworthy data, pickers keep pace, and the requirement's real goal is better served than by the original rule.

Trade-offs and pitfalls

• Siding with whoever signs the contract buys the deal and loses the renewal. Siding with users while ignoring the buyer ignores the pressure to justify spend.
• Saying yes to a custom version for one buyer creates a fork (a separate customer-specific version of the product) you maintain for years. Prefer a setting that other buyers can use. A segment is a group of similar customers (for example by size or industry), and a setting aimed at a segment spreads the cost of building it.
• What would flip my call: a regulatory or contractual requirement keeps the field, and the work shifts to minimising its burden. A buyer who will not move on a deal outside the segment you serve well may be one to decline.
• Solutions Architects can often bridge in this deal with configuration and rollout design; Product Managers own whether the pattern becomes a roadmap item.

Direct answer

Anecdotes tell you where to look, not how big it is. I would set the validation bar before collecting evidence, then use a mix of qualitative and quantitative checks (qualitative means what people say and do in depth, quantitative means counts across many), and treat each result as stronger or weaker according to what the person did, not what they said. A pain is worth acting on when it is frequent enough, severe enough, hits enough of the target group and is poorly served by what they do today.

Terms used below

• Problem interview: a conversation about the customer's past experience of a problem, not a pitch of your solution.
• Representative sample: respondents who resemble the whole target group, not just the friendly or vocal ones.
• Silent churners: customers who leave without ever complaining, so they never appear in tickets.
• Prevalence: how common the problem is across the whole group.
• Smoke test: a cheap fake offer (such as a sign-up page for a feature that does not exist yet) to see who responds.
• Commitment test: asking for something real in return, such as time, money or an introduction.
• Leading question: a question that suggests the answer you want.

Set the bar first (examples to be tuned)

For instance: at least 6 of 8 interviewees from the target segment describe a specific recent incident without prompting, tickets mentioning the problem come from at least 3% of accounts (illustrative), and at least a quarter of surveyed users report it at least monthly, and some people have already paid in time, money or a workaround. Writing this down first stops us from reading support for our idea into whatever we find.

Six checks, what counts as validation, and where each misleads

Leading versus good interview questions

• Leading: "Wouldn't it be great if you could find invoices instantly?" (it hands over the answer, and people politely agree).
• Good: "Tell me about the last time you needed an old invoice. What did you do first? What happened next? What did it cost you?" Follow with "Why?" or "Can you show me?" to turn a vague answer into a story.

Signal strength

Opinions are weakest, stated intent is better, observed behavior is stronger, and commitment is the strongest. For B2B (selling to businesses), test willingness to switch vendors: ask what they would stop using, who must approve, when the current contract ends and what migration costs them. "We would love that" is cheap; "send me a pilot agreement" is not.

Worked example (illustrative)

Three account managers say customers cannot find invoices. The team tags tickets and finds 120 of 4,000 last quarter, or 3.0% of tickets, mention invoices. Counting distinct accounts, as the bar requires, those tickets come from 96 of 2,000 accounts, or 4.8% (illustrative), so the account bar of 3% is cleared. The team also interviews 8 customers across three sizes. Six describe losing time at month end. One says finance already built a spreadsheet to track invoices. Result: real for a defined segment, not yet proven broad. The next step is a commitment check with the segment that feels it most.

Pitfall

Declaring validation after several friendly interviews. Small samples find themes, not prevalence.

Direct answer

I would tell one specific story: where I watched a user, what I saw, why the data could not have shown it, and what I changed because of it. The best stories show the firsthand observation changed a decision, not just my opinion.

How to structure the story

1. Setting: which channel (an interview, a support call, or shadowing, which means sitting in on a customer doing real work without steering them), and why you were there.
2. The observation: what the user did or said, in concrete detail.
3. Why the data missed it: the gap, usually that the behavior happened outside the product or the metric measured the wrong thing.
4. What I did: verification with more users, a clear problem statement, and a change.
5. Result and caution: what happened, and how I kept one visit from becoming a false certainty.

Worked example (an illustrative story)

Product analytics said the export step had a high success rate: people clicked export and got their file. In a call a customer-success colleague invited me to join, I watched a user export a report, open a spreadsheet, spend a long stretch reformatting columns, and paste a summary into a slide for their manager.

The data could not show this because it all happened outside the product, and "export succeeded" hid the real job: producing something a manager can read. I asked three more customers for a 20-minute screen share of their last export-to-report workflow. Two showed the same pattern, which I treated as a direction, not proof. I wrote the problem statement as "users need a manager-ready summary, not raw data" and brought a short clip to planning.

We shipped a small scheduled summary with the four columns people kept. The real test was behavior afterwards: whether those customers still reformatted by hand, which I checked in follow-up conversations, and whether they kept sharing the summary. Illustrative result: of the 12 customers who received the summary in the first month, 9 stopped reformatting by hand when I asked them about it three weeks later, and 7 were still forwarding the summary to their managers, so I kept it and added the fifth column the other three asked for. The caution: this is a dozen customers who were willing to talk to me, so I read it as encouraging, not as proof for the whole base.

Pitfalls

• A story where the data and the visit agreed has little to teach. Pick one where the visit added something.
• "I always talk to users" without a specific observation and a specific change sounds rehearsed.
• Over-reading one visit. Say how you checked it.

Direct answer

I would not open with "we should delay". I would make the case in revenue and risk terms, show options each priced in days, recommend the smallest change that protects the revenue date, and ask for a staged rollout with a pre-agreed trigger. If I lose, I commit to the decision, instrument it, and reopen it only on agreed data.

Step 1: make sure the finding is solid

Usability testing means watching real users attempt tasks. Check how many participants failed, on which task, and whether they were blocked or only slowed. Illustrative finding used below: 5 of 8 participants could not find where to enter the promotion code at checkout. Five failures out of eight is a strong signal but a small sample, so confirm with analytics (the funnel drop-off, meaning the share of users who quit at each step of the flow, at that step) or a quick unmoderated test (participants do the task alone, with their screen recorded and no facilitator, so you can add many more of them cheaply).

Step 2: translate it into leadership's currency

Leadership is optimizing for revenue, so express confusion as lost revenue. Illustrative numbers: 10,000 users start the flow each month, baseline completion (today's share of starters who finish) is 70%, and beta data suggests completion could fall to 62% (an estimate to be confirmed, not a measurement).

Why the delay costs money rather than just moving it: the promo runs in a fixed window, so each day the launch slips is a day of promo revenue that does not happen later (illustrative: $48,000 a month, so 7/30 of it for a week). The two costs have different shapes. The delay is paid once ($11,200). The confusion is paid every month until it is fixed ($16,000 a month), so a one-week fix costs less than a single month of confusion and the gap widens each month the problem stays.

Step 3: the one-slide executive framing

One slide with the decision wanted as the title, a 20-second clip next to the number as evidence, three options with days and revenue effect, and a recommendation with a tripwire (a pre-agreed trigger that reverses or pauses the plan).

What the fix in option C is: relabel the small "Have a code?" link as a visible "Promo code" field and place it above the pay button. It changes wording and position only, not payment logic, so it is cheap (illustrative: 2 to 3 days of work), which is why it fits inside the launch date. The 10% of users who get it are a cohort (a group of users followed together), and the other 90% stay on the old layout until the numbers say it is safe to widen.

Recommendation: C. Tripwire: pause the promo placement if completion stays below 65% across the first 300 starters in the 10% cohort, which separates the estimated 62% from the 70% baseline. It flips to B if the 10% cohort breaches that line. Why 300 starters and not three days: 10% of 10,000 monthly starters is about 1,000 a month, roughly 33 a day, so three days is only about 100 starters, where completion is measured to within roughly 4.8 points (standard error), wider than the 3-point gap between 62% and 65%, so a three-day read would fire or stay silent largely by chance. 300 starters (about nine days) cuts that to roughly 2.8 points, which makes it a screen, not a verdict. A firm read of 62% against the 65% line needs about 700 starters (about three weeks; this is a one-sided 95% read, where the 3-point gap equals 1.64 standard errors, and a conventional two-sided 95% read would need about 1,000 starters, roughly a month at 10%), so if the date allows, widen the cohort to 30% to get there in about a week. Assign the cohort at random so the comparison with the other 90% is fair.

If I lose

Commit and ship without sandbagging (quietly half-hearted execution that lets the decision you opposed fail). Put in the instrumentation (the tracking that records which steps users complete, so the result is visible), log the fix in the backlog with an owner and a date, share the data at the agreed checkpoint, and escalate only if the tripwire is hit on data everyone accepted. Do not say "I told you so".

The same method in other versions of this fight

The same move (price the harm in leadership's units, offer a smaller option) carries over, with illustrative numbers:

Pitfalls

• An ultimatum ("it cannot ship") ends the conversation; options keep it going.
• Do not quote an estimated 62% as fact. Label it, then use the staged rollout to measure it.
• Delay is not the only lever. Changing scope, placement or rollout size often protects both goals.