Customer and User Obsession Questions

Direct answer

I name the specific harm and who it falls on, translate it into a risk the business recognises, bring a workable alternative instead of only a "no", and involve whoever owns the risk (security, privacy or legal). If I have not faced it, this is the exact sequence I would follow. I do not trade it off with revenue silently; the decision gets made visibly by someone with authority.

Method

1. State the harm concretely. "Showing a user's location to people they have blocked" is a harm. "Privacy concerns" is not.
2. Weigh severity, not only share. A small vulnerable segment (people at risk of stalking or harassment, for instance) can matter more than its percentage suggests because the harm is serious and hard to undo.
3. Frame the risk: severity, likelihood, reversibility (can it be undone? a shared location or a sent notification cannot be taken back, while a button colour can), and the cost to trust (users who learn their contact details exposed someone may leave and tell others). For legal specifics, bring in counsel instead of guessing.
4. Quantify the exposure and the alternative. Sizing the people affected makes it concrete.
5. Offer options: ship with a safer default, ship opt-in, delay a week for a fix, or ship narrower.
6. Escalate with a record, not by going around people.

Worked example (illustrative): contact syncing by default

A team plans to turn on contact syncing by default (default-on: switched on for everyone until they turn it off, versus opt-in: off until the user chooses to turn it on) because it lifts sign-ups. With 200,000 monthly active users (MAU), if 2% belong to a group where being found by contacts is dangerous, that is 4,000 people (2% of 200,000). The analytics lead estimates the default-on version brings about 5,500 sign-ups a month against about 5,000 for opt-in, roughly 500 extra a month. That is the revenue argument, and it has to be weighed against 4,000 existing people exposed on day one in a way that cannot be undone.

The pushback, in words I might use in the meeting: "If our estimate holds that about 2% of our 200,000 monthly users, so roughly 4,000 people, are in situations where being found through a phone contact list could put them in danger (for example someone hiding from an abusive ex-partner), then default-on exposes all of them on day one, and a notification to a contact cannot be unsent. Opt-in costs us some of the roughly 500 extra monthly sign-ups, and I would like to measure how much for a week before we decide." The option I bring: opt-in with a clear prompt shown at the moment of value, plus a one-week measured test of how much of the lift survives. The outcome is a trade the team makes knowingly, with a rollback condition and an owner. If leadership still chooses the risky default, I would record my objection and ask that the privacy review (the privacy team's formal check) sign off in writing.

Pitfalls

• Invoking safety as a trump card without evidence. It loses credibility.
• Being the lone "no". Bring an alternative and an ally.
• Skipping the record: if it goes wrong, the paper trail protects users and you.
• Not-yet-faced answer: be explicit that you are describing a plan, and still give specifics.

Direct answer

Choose a moment when you had already committed (designed it, built toward it, or announced it) and something you watched users do contradicted the plan. Say what you saw, show that you checked it was not a one-off, change course openly, and prove the change helped. The strongest stories name the sunk cost (work already spent) and still pivot, because the evidence was specific and repeated.

Story shape (Situation, Task, Action, Result)

• Situation: the direction, why it was reasonable, and your personal stake in it.
• Task: the learning moment. Name the source: a usability session (watching someone try the product while they think aloud), support tickets, or usage data.
• Action: three beats. (1) Describe behaviour you observed, not opinions. (2) Look for a second source before acting. (3) Tell the team what is being cut, what is reused, and why.
• Result: one measurable outcome plus what you would do differently.

Worked example (illustrative skeleton; substitute your own real story)

A product designer and an engineer are three weeks into a step-by-step setup wizard for a reporting tool, assuming new customers are lost and need hand-holding. In moderated sessions (a researcher or colleague is present, giving tasks and asking questions live) with six new customers, every one pauses at step one and asks whether they can just bring their existing spreadsheet. Two give up. Six sessions cannot show how common this is, so the next step is to look at support tickets tagged "setup" (same request repeated) and the sign-up funnel (the steps from starting sign-up to first success, showing where people drop off), where roughly half of sign-ups stop at step one (illustrative).

With three sources agreeing, the designer writes a one-page note: the original assumption, what was observed, work already spent, and a replacement (spreadsheet import with a preview, no wizard). An excerpt:

Assumption: new customers need a guided setup wizard.
Observed:   6 of 6 asked at step one to import their spreadsheet; 2 of 6 gave up.
            9 of the last 40 setup tickets ask the same thing (illustrative).
Spent:      3 weeks. The progress component is reusable.
Proposal:   replace the wizard with spreadsheet import plus preview; cut its five
            remaining steps.
Check:      re-test with the same 6 people in 2 days, then track first report
            within 24 hours.

The wizard's progress component is reused for the import steps, so part of the sunk work survives. A clickable prototype (linked mock screens people can tap through, with no working product behind them) goes to the same participants within two days.

Result to report: the share of new accounts that reach their first report within 24 hours, compared across two equal-length windows before and after. A result might sound like this (illustrative): "In the four weeks before, 31% of new accounts got a first report within 24 hours (62 of 200). In the four weeks after, 48% (96 of 200). A marketing push ran in week three, so I also checked sign-ups from the same channel, and the rise held." In your own story, state your real two numbers and the caveat that other changes shipped in the same period.

Trade-offs and pitfalls

• One loud user is not evidence. Cross-check with a second and third source.
• Sunk cost: say the cost out loud ("we had spent three weeks") so the pivot reads as a decision, not an accident.
• Do not make the story about being right. Credit the users and the colleagues who adapted.
• Say what you would do differently: test a rough prototype in week one, before building.
• Engineering variant: a machine learning engineer who tuned a model for accuracy, then learned users cared about response time and explainability, tells the same story with the metric changed.

Direct answer

I would ask rarely, at the right moment, of a sample, and give people an always-available way to volunteer feedback, then compare the two so loud, angry voices do not stand for everyone. What comes back is tagged, linked to usage context, counted by theme, and answered, so people see that it matters.

Design: when to ask

• At a natural completion point, such as right after a user finishes a task, not mid-task. Never interrupt checkout, error recovery or a first session.
• Eligibility rules (the conditions a user must meet before being shown the prompt): only users who have done the relevant thing a few times, and suppress (hide the prompt from) anyone who dismissed a prompt recently.
• Frequency cap (a limit on how often the same user is asked): one prompt per user per fixed window (for example 60 days, a design choice to tune).
• Sample, do not blast: show the prompt to a random slice of eligible users.

Illustrative sizing: with 20,000 eligible users a week and a 10% sample, 2,000 see the prompt. At an assumed 10% response rate (responses divided by users shown the prompt), that is about 200 answers a week.

Design: how to ask

• One quick question (a rating or thumbs up or down) plus an optional free-text box.
• Dismissible in one tap, and the dismissal is remembered.
• Capture context automatically (page, app version, plan, session ID) with clear consent, so users do not have to describe where they were.

Example (illustrative): right after a user exports a report successfully, a small card appears at the bottom of the screen, not a blocking pop-up. It reads "How was exporting that report?" with thumbs up and thumbs down. A tap opens an optional box, "Anything we should know?", and a "Not now" link closes the card for 60 days. The question is about the task just finished, not about the product in general.

Avoiding the angriest-only problem

A permanent "send feedback" button draws people with strong feelings, usually bad ones. A sampled prompt at a moment of success reaches satisfied and neutral users too. Keep both channels, label them in reports, and compare responders with the user base (plan, tenure, usage) to see who is missing.

Variants

For AI assistants, route by severity: harmful or high-stakes errors escalate immediately, ordinary quality complaints go into the weekly theme review.

What to do with the responses

1. Tag by theme, attach usage context, and count distinct users per theme.
2. Weekly review: urgent issues to engineering, repeated themes to the backlog, how-to questions to docs.
3. Reply to people who left contact details, and tell them when it ships.
4. Track the health of the channel itself: response rate by segment and share of respondents who are negative versus the base's behaviour.

Pitfalls

Over-asking trains users to dismiss the prompt. Treating the average rating as a verdict hides who answered. Collecting feedback and never replying teaches users to stop giving it.

Direct answer

Rank by estimated customer pain per unit of effort, and build pain from independent evidence: how many people hit the problem (reach), how badly (severity), and how much work the fix takes (effort). Loudness is controlled by counting distinct affected users from behaviour data instead of counting messages, and the ranking is checked against qualitative evidence from interviews, support transcripts and Net Promoter Score (NPS) comments. NPS is a survey asking customers how likely they are to recommend you on a 0 to 10 scale, with a free-text reason.

Step 1: deduplicate and tag

Cluster the 150 reports into distinct underlying problems, then tag each with the area, the symptom and a severity class: 3 = loses data or blocks a core task, 2 = slows a task but a workaround exists, 1 = annoyance or cosmetic.

Step 2: measure reach from behaviour, not volume

Use telemetry (usage data the product records automatically): error logs, funnel drop-offs (the share of users who quit at each step of a multi-step flow such as checkout), usage events. Tickets overcount persistent complainers and undercount quiet sufferers. One account sending nine emails is one account. Upvotes (votes from users on a public request) are volume too, so use them only as a hint, not as reach.

Step 3: score

priority = reach (% of active users) x severity (1 to 3) / effort (engineer-days)

Enter reach as a plain number of percentage points (12% goes in as 12, 0.4% as 0.4, never 0.12 or 0.004). Plain English: more people, hurt more, for less work, ranks higher. Treat scores in the same ballpark as ties and break them with judgement.

Step 4: let qualitative evidence change the ranking (illustrative numbers)

Tag each interview, NPS comment and transcript to the same issue clusters, with explicit rules: a transcript showing lost work or abandonment raises severity to 3; one showing a trivial workaround lowers it. Record the supporting quote so the change can be audited.

Before the transcripts, C outranks B. Reading five transcripts shows users redoing hours of work after failed exports, so B moves to severity 3 and overtakes C. The loudest report, A, is cosmetic (severity 1 on the Step 1 scale) and stays last.

Conflicting requests from strategic customers

Strategic customers are the accounts the company has named as most important to win or keep. Do not override the score silently. Reserve a small explicit quota, agreed in advance, for documented churn risk (a customer likely to cancel), and judge those items on evidence (renewal date, stated blocker). Everything else competes on the score.

Checking the ranking reflects real pain

• Spot-check with a few users from the top and the bottom of the list.
• After fixing the top items, confirm tickets and funnel drop-offs for those issues fell.
• Compare who is complaining with who is affected in the telemetry.

Variation: complaints about model quality (ML-powered features)

The same scoring still applies. For a feature powered by a machine learning model, add these checks in the first 24 hours: (1) did complaints start after a model, data or prompt release (a change to the instructions given to a language model)? (2) replay saved examples to tell a regression (something that used to work and now does not) from an old weakness; (3) compare today's inputs with the training period (data drift: real inputs shift away from what the model learned); (4) rule out the product around the model (truncated text, meaning input cut off before it reaches the model; timeouts; a stale cache, meaning an old saved answer served instead of a fresh one) before blaming the model; (5) check whether complaints cluster in a segment.

Example (illustrative): 40 tickets say "the assistant summaries are wrong since last week". Check 1 shows all 40 began within two days of a prompt release. Check 2 replays 50 saved inputs and finds the new prompt drops the last paragraph of long documents, so it is a regression. Check 4 shows the same truncation in a log. The fix is to roll back the prompt, and check 2 is what caught it.

Variation: player-value scoring (games)

Weight reach by player value, for example new players (whose retention, meaning whether they keep playing, decides the game's future) versus long-tenured or paying players, as a capped multiplier so it does not recreate a revenue-ranked skew.

Example (illustrative): a bug hits 3% of players, severity 2, effort 4, so the base score is 3 x 2 / 4 = 1.5. If most affected players are paying, apply a multiplier of 1.5 (the cap): 1.5 x 1.5 = 2.25. Uncapped, a spending-weighted multiplier of 5 would give 7.5 and let paying players outrank every other issue, which is the skew the cap prevents.

Direct answer

I decide by how costly and reversible being wrong is, then shrink the validation to fit the deadline instead of choosing between "validate fully" and "skip". Usually the answer is: validate the riskiest assumption quickly, ship in a reversible way, and watch real behaviour after release.

Decision steps

1. Name the riskiest assumption ("users will understand the new permission screen").
2. Ask: if we are wrong, how bad, and how fast can we undo it? Wrong on a colour is cheap. Wrong on billing or data handling is not.
3. Pick the smallest check that touches that assumption: five short sessions with a clickable prototype, a fake-door test (a button for a feature that does not exist yet, to count clicks), or a survey of ten existing customers.
4. Ship in a way you can reverse: a feature flag (a switch that turns the feature on or off without a new release) and a staged rollout (a small share of users first).
5. Pre-agree the stop signal: "If task completion or error rate on this screen is worse than the old version, we switch it off."

Worked example (illustrative)

A three-week deadline (15 working days) for a new checkout flow. The riskiest assumption is that users will find the new "choose delivery date" step. Spend three days (20% of the schedule) on a prototype test with five existing customers, and fix the two points where they hesitate. Release to 10% of traffic behind a flag in week three, compare checkout completion against the old flow for a few days, then widen. The stop signal is agreed before the release, in numbers (illustrative): if checkout completion in the 10% group is more than 2 percentage points below the old flow over three full days, or the share who abandon on the delivery-date step rises noticeably above the old flow's, the flag is switched off and the old flow returns, owned by the engineer on call. Only if neither trips do we widen to 50%, then 100%. The 2-point rule only works with enough volume: at a 30% baseline completion rate, detecting a 2-point drop with 80% power at the usual 5% significance level needs roughly 8,000 orders in each group (about 9,800 at a 50% baseline; computed with the standard two-proportion formula). If three days at 10% traffic will not produce that many orders, either run the 10% stage longer, treat the rule as a guard against large drops only (for example 5 points or more), or watch the delivery-date step abandonment rate, which moves faster. Say which in the story, so the stop signal is not tripped or missed by noise.

What I tell stakeholders: "We are not skipping validation. We are doing the part that can change our decision, in three days, and we have an off switch for the rest."

Pitfalls

• Treating validation as a gate that always takes weeks. Match effort to risk.
• Treating the deadline as sacred without asking who set it and what really breaks if it slips.
• Validating the wrong thing: easy to test is not the same as risky.
• No stop signal: a rollout without a pre-agreed trigger is just shipping.