Customer and User Obsession Questions

Direct answer

I decide by how costly and reversible being wrong is, then shrink the validation to fit the deadline instead of choosing between "validate fully" and "skip". Usually the answer is: validate the riskiest assumption quickly, ship in a reversible way, and watch real behaviour after release.

Decision steps

1. Name the riskiest assumption ("users will understand the new permission screen").
2. Ask: if we are wrong, how bad, and how fast can we undo it? Wrong on a colour is cheap. Wrong on billing or data handling is not.
3. Pick the smallest check that touches that assumption: five short sessions with a clickable prototype, a fake-door test (a button for a feature that does not exist yet, to count clicks), or a survey of ten existing customers.
4. Ship in a way you can reverse: a feature flag (a switch that turns the feature on or off without a new release) and a staged rollout (a small share of users first).
5. Pre-agree the stop signal: "If task completion or error rate on this screen is worse than the old version, we switch it off."

Worked example (illustrative)

A three-week deadline (15 working days) for a new checkout flow. The riskiest assumption is that users will find the new "choose delivery date" step. Spend three days (20% of the schedule) on a prototype test with five existing customers, and fix the two points where they hesitate. Release to 10% of traffic behind a flag in week three, compare checkout completion against the old flow for a few days, then widen. The stop signal is agreed before the release, in numbers (illustrative): if checkout completion in the 10% group is more than 2 percentage points below the old flow over three full days, or the share who abandon on the delivery-date step rises noticeably above the old flow's, the flag is switched off and the old flow returns, owned by the engineer on call. Only if neither trips do we widen to 50%, then 100%. The 2-point rule only works with enough volume: at a 30% baseline completion rate, detecting a 2-point drop with 80% power at the usual 5% significance level needs roughly 8,000 orders in each group (about 9,800 at a 50% baseline; computed with the standard two-proportion formula). If three days at 10% traffic will not produce that many orders, either run the 10% stage longer, treat the rule as a guard against large drops only (for example 5 points or more), or watch the delivery-date step abandonment rate, which moves faster. Say which in the story, so the stop signal is not tripped or missed by noise.

What I tell stakeholders: "We are not skipping validation. We are doing the part that can change our decision, in three days, and we have an off switch for the rest."

Pitfalls

• Treating validation as a gate that always takes weeks. Match effort to risk.
• Treating the deadline as sacred without asking who set it and what really breaks if it slips.
• Validating the wrong thing: easy to test is not the same as risky.
• No stop signal: a rollout without a pre-agreed trigger is just shipping.

Direct answer

I would build a small weekly habit of looking directly at real user behavior and words, using the cheapest channels, and I would track a decision log so I can see whether that evidence changes what the team builds. Closeness only counts if it moves a decision.

A week-to-week routine (about 75 minutes weekly, plus a 45-minute call every other week)

Check your company's privacy rules before watching recordings or joining calls.

How to tell whether it changes decisions

Keep a one-line decision log: decision, the user evidence behind it, and whether the evidence changed the plan. Each quarter, count the decisions where user evidence changed the scope, the order or the answer. If that count stays at zero for a quarter, the habit is a ritual.

Worked example (illustrative)

A data engineer owns a pipeline that feeds a dashboard. Ticket reading shows several users saying "the numbers are old at 9am". Shadowing a call shows managers opening the dashboard for a morning meeting before the refresh finishes. The team's plan was to make the job 20% faster. The evidence changed it: move the refresh earlier and show a visible "last updated" time. That one decision goes in the log as evidence-changed.

Pitfalls and cost

• Sampling bias: your sample leaves out some kinds of users, so it misleads you. Loud tickets are not typical, so sample randomly.
• Replays show what, not why. Pair them with tickets or a call.
• Privacy: mask personal data and follow policy.
• Time cost: under two hours a week on average, so protect it on the calendar or it vanishes.
• Ritual drift: reading without a log produces warm feelings and no change.

Direct answer

I would write the assumption as a claim data could contradict (falsifiable: able to be shown wrong), set the stop bar before looking at anything, and use evidence already in hand (logs, tickets, past experiments) plus one light behavioral probe instead of calls. Two weeks is enough: days 1 to 2 for the claim and the bar, days 3 to 7 for existing data, days 8 to 12 for a probe if the data is not decisive, and days 13 to 14 to decide.

Elaboration

1. Make it falsifiable. "Users want better results" cannot fail. "Users who rewrite a query within 60 seconds do so because the first results were poor" can.
2. Set the bar from the size of the prize. Ask how large the affected group must be to justify the work, and write it down first. Otherwise any result gets argued into a yes.
3. Look at behavior you already have: logs, tickets, earlier experiments.
4. Probe if needed (a behavioral probe: a cheap test that watches what users do, not what they say): a fake-door button (a button for a feature that does not exist yet, to count clicks) or a small in-product prompt.
5. Decide in three zones: clearly above the bar, go; clearly below it, stop or redesign; between, run the probe.

Worked example (illustrative; retraining a search ranker)

Assumption: many users reformulate their query (retype it with different words) because the ranker (the model that orders search results) put the useful result too low, and retraining it (re-running its learning on fresh data) will fix it.

Bar: the pattern must appear in at least 10% of sessions. Reasoning: retraining will not rescue every case, because some reformulations come from vague wording or missing content that no ranking change fixes. Taking an optimistic guess that it fixes at best half, 10% of sessions becomes about 5% of sessions improved, the smallest gain the team judged worth about two months of work.

Test: sample 2,000 sessions from the logs. A session matches if the user reformulated within 60 seconds and then clicked a result that the first list had placed lower. Suppose 130 match.

How to read the margin: 130 matches in 2,000 sessions is one sample, and a different random 2,000 would give a slightly different rate. The margin says how far the true rate is plausibly from 6.5%. The 95% means that if you repeated this sampling many times, about 95 of 100 intervals built this way would contain the true rate. The 1.96 is the standard multiplier for 95%, and the square-root part is the typical wobble of a rate measured on 2,000 sessions (the normal approximation, a textbook shortcut that works well when there are many matches and many non-matches).

The whole interval is under 10%, so the stop holds even allowing for sampling noise, assuming the sample is random. I would stop, and also read 20 raw sessions to see what is going on, because logs show what happened, not why.

Pitfalls

• Choosing the bar after seeing the number.
• A narrow sample window, such as one holiday week, that hides the usual behavior.
• Treating "no calls" as "no evidence". Behavioral data is evidence, just thin on motive.

Direct answer

I would not decide on the size of the segment alone. I would compare expected value per unit of effort (the benefit I expect from each option, divided by what it costs to build) for each direction, check that the numbers rest on evidence, and weigh the strategic risks. For typical assumptions, I lean toward a thin version of the premium feature first, validated with a few high-value users, because their need is stronger and a small accuracy gain may not change their behaviour. I would reverse that if the premium group's pain turns out to be unvalidated or if accuracy problems are driving churn across all users.

Decision frame

1. Define the groups. "High-value" must mean something concrete such as revenue or strategic importance. A small effect on 90% is not automatically bigger than a large effect on 10%.
2. Estimate benefit per group: how many people, how much their behaviour (retention, meaning the share of users who stay; upgrade; usage) changes, and what each is worth.
3. Estimate cost and risk: engineering effort, time to ship, data needed for personalization, privacy, and the chance each effect is smaller than hoped.
4. Strategic factors: dependence on a concentrated group (losing a few of them hurts a lot), what competitors are offering them, and whether accuracy is a baseline that everyone expects.
5. Test cheaply before building fully. A prototype or hand-run pilot for the 10% group, and a check whether an accuracy gain actually moves retention for the 90%.

Worked example (illustrative numbers, not real data)

Assume 1,000,000 users: 900,000 standard users worth $10 a year each, and 100,000 high-value users worth $60 a year each.

On these assumptions the premium feature is worth about 6.7 times more per year ($300,000 divided by $45,000). The decision then depends on cost and on whether the assumed effects are real. To compare per unit of effort, assume the accuracy work takes 2 person-months and the premium feature 10 (five times as much). Accuracy: $45,000 / 2 = $22,500 per person-month. Premium: $300,000 / 10 = $30,000 per person-month. The gap narrows from 6.7 times to about 1.3 times (30,000 / 22,500), so the choice now hinges on how far you trust the 5-point assumption. And if the 5-point effect is only 1 point, premium gives $60,000 a year, which looks close to the accuracy option's $45,000 in total value but is not close per unit of effort: $60,000 / 10 = $6,000 per person-month against $22,500 for accuracy, about 3.75 times worse. Total value alone would hide that.

Recommendation

Run a small premium pilot with a handful of high-value customers, measure the retention or usage effect it produces, and keep a modest accuracy workstream going so the 90% do not drift down. Commit the larger investment once the pilot shows the effect.

Pitfalls

Do not treat every user as equal, and do not treat revenue as the whole story. Do not rely on assumed effects without a pilot. Watch for building something only a few love at the cost of the quality everyone depends on.

Direct answer

I name the specific harm and who it falls on, translate it into a risk the business recognises, bring a workable alternative instead of only a "no", and involve whoever owns the risk (security, privacy or legal). If I have not faced it, this is the exact sequence I would follow. I do not trade it off with revenue silently; the decision gets made visibly by someone with authority.

Method

1. State the harm concretely. "Showing a user's location to people they have blocked" is a harm. "Privacy concerns" is not.
2. Weigh severity, not only share. A small vulnerable segment (people at risk of stalking or harassment, for instance) can matter more than its percentage suggests because the harm is serious and hard to undo.
3. Frame the risk: severity, likelihood, reversibility (can it be undone? a shared location or a sent notification cannot be taken back, while a button colour can), and the cost to trust (users who learn their contact details exposed someone may leave and tell others). For legal specifics, bring in counsel instead of guessing.
4. Quantify the exposure and the alternative. Sizing the people affected makes it concrete.
5. Offer options: ship with a safer default, ship opt-in, delay a week for a fix, or ship narrower.
6. Escalate with a record, not by going around people.

Worked example (illustrative): contact syncing by default

A team plans to turn on contact syncing by default (default-on: switched on for everyone until they turn it off, versus opt-in: off until the user chooses to turn it on) because it lifts sign-ups. With 200,000 monthly active users (MAU), if 2% belong to a group where being found by contacts is dangerous, that is 4,000 people (2% of 200,000). The analytics lead estimates the default-on version brings about 5,500 sign-ups a month against about 5,000 for opt-in, roughly 500 extra a month. That is the revenue argument, and it has to be weighed against 4,000 existing people exposed on day one in a way that cannot be undone.

The pushback, in words I might use in the meeting: "If our estimate holds that about 2% of our 200,000 monthly users, so roughly 4,000 people, are in situations where being found through a phone contact list could put them in danger (for example someone hiding from an abusive ex-partner), then default-on exposes all of them on day one, and a notification to a contact cannot be unsent. Opt-in costs us some of the roughly 500 extra monthly sign-ups, and I would like to measure how much for a week before we decide." The option I bring: opt-in with a clear prompt shown at the moment of value, plus a one-week measured test of how much of the lift survives. The outcome is a trade the team makes knowingly, with a rollback condition and an owner. If leadership still chooses the risky default, I would record my objection and ask that the privacy review (the privacy team's formal check) sign off in writing.

Pitfalls

• Invoking safety as a trump card without evidence. It loses credibility.
• Being the lone "no". Bring an alternative and an ally.
• Skipping the record: if it goes wrong, the paper trail protects users and you.
• Not-yet-faced answer: be explicit that you are describing a plan, and still give specifics.