Customer and User Obsession Questions

Direct answer

Rank by estimated customer pain per unit of effort, and build pain from independent evidence: how many people hit the problem (reach), how badly (severity), and how much work the fix takes (effort). Loudness is controlled by counting distinct affected users from behaviour data instead of counting messages, and the ranking is checked against qualitative evidence from interviews, support transcripts and Net Promoter Score (NPS) comments. NPS is a survey asking customers how likely they are to recommend you on a 0 to 10 scale, with a free-text reason.

Step 1: deduplicate and tag

Cluster the 150 reports into distinct underlying problems, then tag each with the area, the symptom and a severity class: 3 = loses data or blocks a core task, 2 = slows a task but a workaround exists, 1 = annoyance or cosmetic.

Step 2: measure reach from behaviour, not volume

Use telemetry (usage data the product records automatically): error logs, funnel drop-offs (the share of users who quit at each step of a multi-step flow such as checkout), usage events. Tickets overcount persistent complainers and undercount quiet sufferers. One account sending nine emails is one account. Upvotes (votes from users on a public request) are volume too, so use them only as a hint, not as reach.

Step 3: score

priority = reach (% of active users) x severity (1 to 3) / effort (engineer-days)

Enter reach as a plain number of percentage points (12% goes in as 12, 0.4% as 0.4, never 0.12 or 0.004). Plain English: more people, hurt more, for less work, ranks higher. Treat scores in the same ballpark as ties and break them with judgement.

Step 4: let qualitative evidence change the ranking (illustrative numbers)

Tag each interview, NPS comment and transcript to the same issue clusters, with explicit rules: a transcript showing lost work or abandonment raises severity to 3; one showing a trivial workaround lowers it. Record the supporting quote so the change can be audited.

Before the transcripts, C outranks B. Reading five transcripts shows users redoing hours of work after failed exports, so B moves to severity 3 and overtakes C. The loudest report, A, is cosmetic (severity 1 on the Step 1 scale) and stays last.

Conflicting requests from strategic customers

Strategic customers are the accounts the company has named as most important to win or keep. Do not override the score silently. Reserve a small explicit quota, agreed in advance, for documented churn risk (a customer likely to cancel), and judge those items on evidence (renewal date, stated blocker). Everything else competes on the score.

Checking the ranking reflects real pain

• Spot-check with a few users from the top and the bottom of the list.
• After fixing the top items, confirm tickets and funnel drop-offs for those issues fell.
• Compare who is complaining with who is affected in the telemetry.

Variation: complaints about model quality (ML-powered features)

The same scoring still applies. For a feature powered by a machine learning model, add these checks in the first 24 hours: (1) did complaints start after a model, data or prompt release (a change to the instructions given to a language model)? (2) replay saved examples to tell a regression (something that used to work and now does not) from an old weakness; (3) compare today's inputs with the training period (data drift: real inputs shift away from what the model learned); (4) rule out the product around the model (truncated text, meaning input cut off before it reaches the model; timeouts; a stale cache, meaning an old saved answer served instead of a fresh one) before blaming the model; (5) check whether complaints cluster in a segment.

Example (illustrative): 40 tickets say "the assistant summaries are wrong since last week". Check 1 shows all 40 began within two days of a prompt release. Check 2 replays 50 saved inputs and finds the new prompt drops the last paragraph of long documents, so it is a regression. Check 4 shows the same truncation in a log. The fix is to roll back the prompt, and check 2 is what caught it.

Variation: player-value scoring (games)

Weight reach by player value, for example new players (whose retention, meaning whether they keep playing, decides the game's future) versus long-tenured or paying players, as a capped multiplier so it does not recreate a revenue-ranked skew.

Example (illustrative): a bug hits 3% of players, severity 2, effort 4, so the base score is 3 x 2 / 4 = 1.5. If most affected players are paying, apply a multiplier of 1.5 (the cap): 1.5 x 1.5 = 2.25. Uncapped, a spending-weighted multiplier of 5 would give 7.5 and let paying players outrank every other issue, which is the skew the cap prevents.

Direct answer

"Search is slow" is a symptom described in the customer's words. To make it actionable I would translate it into a measurable problem statement: who is affected, what they experience, how we measure it, and what good looks like. Then I would test that the measurement explains the complaints, rather than assuming the first reading is right.

Step 1: Quantify the complaints

Tag a sample of comments and tickets by what "slow" means:

• Waiting a long time for results to appear (latency).
• Results appear but are the wrong ones, so the user searches again and again (relevance, which feels slow).
• The page gives no sign it is working, so a short wait feels long (feedback).

Illustrative tagging of 60 comments: 31 describe waiting (a few of these also say the page gave no sign of progress, the third meaning, so a comment can carry more than one tag), 14 say they cannot find things, 15 are unrelated. Then count distinct customers, plan tiers and dates, and note whether they began after a release.

Step 2: Join the complaints to telemetry (the usage and timing data the product records automatically)

For the customers who complained, look at the search response time (use the 95th percentile, the time that 95% of searches beat, as well as the median, the typical middle search), searches per session, and the reformulation rate, the share of searches followed within seconds by a rewritten search. Compare with customers who did not complain. If complainers are mostly accounts with very large data sets, that is a precise lead for engineering.

Step 3: Write the engineering-ready ticket

Problem: Accounts with more than 100,000 records wait too long for search results.
Evidence: 31 of 60 sampled comments mention waiting. For those accounts P95 search
          time is 6.8 s against 1.9 s for other accounts (illustrative numbers).
Metric:   P95 search response time per account tier, measured at the server and
          at the browser when results render. Baseline today: 6.8 s.
Target:   P95 of 3 s or less for the tier (illustrative; the real value is agreed
          with engineering from the measured baseline and user tolerance).
Hypotheses (candidate causes to test): index size; a recent release; query type;
          missing loading state.
Done when: target met for the tier AND complaint volume and reformulation rate fall.

Step 4: How I know it is the real problem

• Reproduce it on a complaining customer's account, or watch a screen recording of their session.
• Check timing: did complaints start when the latency measure worsened? If complaints came first, suspect relevance or perception.
• Check the dose-response (more of the suspected cause should go with more of the effect): do the accounts with the worst measured latency complain most? Illustratively, sort 100 accounts by P95 and cut them into four equal groups. If complaints per 100 users run 1, 2, 4 and 9 from the fastest group to the slowest, the pattern supports latency as the cause. If all four groups sit near 3, latency is not what customers are reacting to.
• After the fix, confirm the complaint rate drops and a behavioural metric (reformulations, abandoned searches) improves. If latency improves but complaints do not, we fixed a different problem.

Trade-offs and pitfalls

• Setting a performance target before measuring. Targets come from baseline and user tolerance, not intuition.
• Handing engineering "customers hate search" with no evidence. They cannot prioritise or verify it.
• Treating ticket volume as a measure of breadth: one customer can file 30 tickets.

Direct answer

A strong answer owns the miss without excuses, shows what I did fast to limit harm (tell the customer early, give an interim fix, coordinate engineering with one owner), what I changed so it would not recur, and how the relationship ended up. The example below is an illustrative skeleton; use your own facts.

Situation

I had told a customer an integration would be ready by their launch date. Two weeks before, it became clear the date was not safe, because engineering found an unexpected limit in a third-party service.

Recovery

1. Tell them early, directly. I called the customer's lead as soon as I knew, rather than waiting for the date to pass. I said what I had promised, what had changed, and that the fault was in how I committed.
2. Offer an interim path. A manual workaround covered their launch while the integration was finished.
3. Coordinate engineering. I became the single point of contact (one person the customer can always reach): one shared channel, a named engineering owner, and a status update to the customer every few days even when nothing had changed.
4. Give a new date with an honest range. I committed only after engineering confirmed it, stating a range and what could move it.

The same story from a reliability seat

If your seat is site reliability engineering, the promise is usually a service level agreement (SLA, the uptime or recovery commitment written into the contract) rather than a launch date, but the recovery has the same shape. Say the service fell below the promised uptime. You post a status update as soon as you know, not after the customer notices. You put in a mitigation (a temporary fix such as failing over to a standby system) while the real fix is built, and you run one incident owner so the customer talks to one person. You then give an honest estimate of the fix, and afterwards run a blameless post-incident review (a written account of what happened and why, aimed at fixing the process, not blaming a person) with named action items such as better alerting or a capacity limit check.

Turning the customer around

The customer was frustrated at first, but the early warning, the workaround and the regular updates gave them what they needed to plan. They later agreed to act as a reference (a customer who vouches for you to prospective customers) for the integration, which is the outcome I aim for: a detractor (an unhappy customer; in the Net Promoter Score survey, someone answering 6 or lower on the 0 to 10 "would you recommend us" question) becoming an advocate because of how a problem was handled.

What I changed afterwards

• I stopped promising dates before engineering confirmed an estimate, and began giving ranges.
• I added a checkpoint at the midpoint of any customer commitment to check confidence.
• I wrote a short review of what went wrong and shared it with the team.

Pitfalls

Blaming engineering or the customer, minimising the miss, claiming a recovery you cannot show, or describing only the apology. The strongest answers show the specific process change. A strong story also shows the specific process change, since an apology alone does not stop a repeat.

Direct answer

I would write the assumption as a claim data could contradict (falsifiable: able to be shown wrong), set the stop bar before looking at anything, and use evidence already in hand (logs, tickets, past experiments) plus one light behavioral probe instead of calls. Two weeks is enough: days 1 to 2 for the claim and the bar, days 3 to 7 for existing data, days 8 to 12 for a probe if the data is not decisive, and days 13 to 14 to decide.

Elaboration

1. Make it falsifiable. "Users want better results" cannot fail. "Users who rewrite a query within 60 seconds do so because the first results were poor" can.
2. Set the bar from the size of the prize. Ask how large the affected group must be to justify the work, and write it down first. Otherwise any result gets argued into a yes.
3. Look at behavior you already have: logs, tickets, earlier experiments.
4. Probe if needed (a behavioral probe: a cheap test that watches what users do, not what they say): a fake-door button (a button for a feature that does not exist yet, to count clicks) or a small in-product prompt.
5. Decide in three zones: clearly above the bar, go; clearly below it, stop or redesign; between, run the probe.

Worked example (illustrative; retraining a search ranker)

Assumption: many users reformulate their query (retype it with different words) because the ranker (the model that orders search results) put the useful result too low, and retraining it (re-running its learning on fresh data) will fix it.

Bar: the pattern must appear in at least 10% of sessions. Reasoning: retraining will not rescue every case, because some reformulations come from vague wording or missing content that no ranking change fixes. Taking an optimistic guess that it fixes at best half, 10% of sessions becomes about 5% of sessions improved, the smallest gain the team judged worth about two months of work.

Test: sample 2,000 sessions from the logs. A session matches if the user reformulated within 60 seconds and then clicked a result that the first list had placed lower. Suppose 130 match.

How to read the margin: 130 matches in 2,000 sessions is one sample, and a different random 2,000 would give a slightly different rate. The margin says how far the true rate is plausibly from 6.5%. The 95% means that if you repeated this sampling many times, about 95 of 100 intervals built this way would contain the true rate. The 1.96 is the standard multiplier for 95%, and the square-root part is the typical wobble of a rate measured on 2,000 sessions (the normal approximation, a textbook shortcut that works well when there are many matches and many non-matches).

The whole interval is under 10%, so the stop holds even allowing for sampling noise, assuming the sample is random. I would stop, and also read 20 raw sessions to see what is going on, because logs show what happened, not why.

Pitfalls

• Choosing the bar after seeing the number.
• A narrow sample window, such as one holiday week, that hides the usual behavior.
• Treating "no calls" as "no evidence". Behavioral data is evidence, just thin on motive.

Direct answer

I would build a small weekly habit of looking directly at real user behavior and words, using the cheapest channels, and I would track a decision log so I can see whether that evidence changes what the team builds. Closeness only counts if it moves a decision.

A week-to-week routine (about 75 minutes weekly, plus a 45-minute call every other week)

Check your company's privacy rules before watching recordings or joining calls.

How to tell whether it changes decisions

Keep a one-line decision log: decision, the user evidence behind it, and whether the evidence changed the plan. Each quarter, count the decisions where user evidence changed the scope, the order or the answer. If that count stays at zero for a quarter, the habit is a ritual.

Worked example (illustrative)

A data engineer owns a pipeline that feeds a dashboard. Ticket reading shows several users saying "the numbers are old at 9am". Shadowing a call shows managers opening the dashboard for a morning meeting before the refresh finishes. The team's plan was to make the job 20% faster. The evidence changed it: move the refresh earlier and show a visible "last updated" time. That one decision goes in the log as evidence-changed.

Pitfalls and cost

• Sampling bias: your sample leaves out some kinds of users, so it misleads you. Loud tickets are not typical, so sample randomly.
• Replays show what, not why. Pair them with tickets or a call.
• Privacy: mask personal data and follow policy.
• Time cost: under two hours a week on average, so protect it on the calendar or it vanishes.
• Ritual drift: reading without a log produces warm feelings and no change.