Customer and User Obsession Questions

Direct answer

I run it as a blameless review (a post-incident review that asks how our process allowed the problem, not who is at fault) that starts with the customer's timeline, not with the question "was our change safe?" Both sides can be right: the change can pass every check we wrote and still break something the customer depended on. The review's job is to find that gap and fix how we change things, not to win the argument.

How I run the review

1. Customer first. Ask the customer to walk through what happened to their business: when they noticed, which workflows broke, what it cost them, and what they did. Capture it in their words in the first section of the review document.
2. Our evidence second. Build a timeline from our side: what changed, when, to whom it rolled out, what notice was given, and what our tests and monitoring saw. Lay the two timelines side by side.
3. Replace "safe" with testable questions. Safe against what? Our contract tests (automated checks that the API still returns what our written spec promises), or the way this customer actually uses the product? Did we know this usage pattern existed? Was the change announced, and could they roll back (return to the old behavior) or opt out (keep the old behavior for themselves)?
4. Include the customer's voice in the room. Invite the customer (or a representative who spoke with them) and read their impact statement aloud before anyone presents engineering findings.
5. Decide fixes at three levels: the repair for this customer, the fix to the change process (notice, a staged rollout that releases to a small share of customers first and widens only if things look healthy, a way to opt out), and the detection gap (what would have alerted us).

Worked example (illustrative)

A list API returns long results in pages, and the default page size is how many records come back per request. We lower that default from 1,000 to 100 records. Our contract tests pass because the API still returns correct data. A customer's nightly export reads only the first page (it never asks for page two, which it never needed under the old default) and has 450 records, so it now silently sees 100, which is 22% of the data (100 of 450). Downstream reports are wrong for days. The product team says the change was safe, and by the written spec it was. The review shows the customer's real usage was undocumented, that the change was announced only in release notes, and that nothing alerted anyone to the truncation. Fixes: a restored option for the old default for this customer, a breaking-change checklist (a pre-release list of questions about who may depend on the old behavior) that asks "who may rely on this behaviour?", advance notice with a date, and a check on response size shifts.

Trade-offs and pitfalls

• Customer-centered does not mean admitting fault before the facts are in. Acknowledge the impact and commit to the review, but let the timeline decide cause.
• Do not let the review end at "customer misused it." Even if true, the question is how we could have known.
• Keep the review about the process and decisions, not individuals, or the engineers will protect themselves instead of sharing facts.

Direct answer

I would tell one specific story: where I watched a user, what I saw, why the data could not have shown it, and what I changed because of it. The best stories show the firsthand observation changed a decision, not just my opinion.

How to structure the story

1. Setting: which channel (an interview, a support call, or shadowing, which means sitting in on a customer doing real work without steering them), and why you were there.
2. The observation: what the user did or said, in concrete detail.
3. Why the data missed it: the gap, usually that the behavior happened outside the product or the metric measured the wrong thing.
4. What I did: verification with more users, a clear problem statement, and a change.
5. Result and caution: what happened, and how I kept one visit from becoming a false certainty.

Worked example (an illustrative story)

Product analytics said the export step had a high success rate: people clicked export and got their file. In a call a customer-success colleague invited me to join, I watched a user export a report, open a spreadsheet, spend a long stretch reformatting columns, and paste a summary into a slide for their manager.

The data could not show this because it all happened outside the product, and "export succeeded" hid the real job: producing something a manager can read. I asked three more customers for a 20-minute screen share of their last export-to-report workflow. Two showed the same pattern, which I treated as a direction, not proof. I wrote the problem statement as "users need a manager-ready summary, not raw data" and brought a short clip to planning.

We shipped a small scheduled summary with the four columns people kept. The real test was behavior afterwards: whether those customers still reformatted by hand, which I checked in follow-up conversations, and whether they kept sharing the summary. Illustrative result: of the 12 customers who received the summary in the first month, 9 stopped reformatting by hand when I asked them about it three weeks later, and 7 were still forwarding the summary to their managers, so I kept it and added the fifth column the other three asked for. The caution: this is a dozen customers who were willing to talk to me, so I read it as encouraging, not as proof for the whole base.

Pitfalls

• A story where the data and the visit agreed has little to teach. Pick one where the visit added something.
• "I always talk to users" without a specific observation and a specific change sounds rehearsed.
• Over-reading one visit. Say how you checked it.

Direct answer

A strong answer owns the miss without excuses, shows what I did fast to limit harm (tell the customer early, give an interim fix, coordinate engineering with one owner), what I changed so it would not recur, and how the relationship ended up. The example below is an illustrative skeleton; use your own facts.

Situation

I had told a customer an integration would be ready by their launch date. Two weeks before, it became clear the date was not safe, because engineering found an unexpected limit in a third-party service.

Recovery

1. Tell them early, directly. I called the customer's lead as soon as I knew, rather than waiting for the date to pass. I said what I had promised, what had changed, and that the fault was in how I committed.
2. Offer an interim path. A manual workaround covered their launch while the integration was finished.
3. Coordinate engineering. I became the single point of contact (one person the customer can always reach): one shared channel, a named engineering owner, and a status update to the customer every few days even when nothing had changed.
4. Give a new date with an honest range. I committed only after engineering confirmed it, stating a range and what could move it.

The same story from a reliability seat

If your seat is site reliability engineering, the promise is usually a service level agreement (SLA, the uptime or recovery commitment written into the contract) rather than a launch date, but the recovery has the same shape. Say the service fell below the promised uptime. You post a status update as soon as you know, not after the customer notices. You put in a mitigation (a temporary fix such as failing over to a standby system) while the real fix is built, and you run one incident owner so the customer talks to one person. You then give an honest estimate of the fix, and afterwards run a blameless post-incident review (a written account of what happened and why, aimed at fixing the process, not blaming a person) with named action items such as better alerting or a capacity limit check.

Turning the customer around

The customer was frustrated at first, but the early warning, the workaround and the regular updates gave them what they needed to plan. They later agreed to act as a reference (a customer who vouches for you to prospective customers) for the integration, which is the outcome I aim for: a detractor (an unhappy customer; in the Net Promoter Score survey, someone answering 6 or lower on the 0 to 10 "would you recommend us" question) becoming an advocate because of how a problem was handled.

What I changed afterwards

• I stopped promising dates before engineering confirmed an estimate, and began giving ranges.
• I added a checkpoint at the midpoint of any customer commitment to check confidence.
• I wrote a short review of what went wrong and shared it with the team.

Pitfalls

Blaming engineering or the customer, minimising the miss, claiming a recovery you cannot show, or describing only the apology. The strongest answers show the specific process change. A strong story also shows the specific process change, since an apology alone does not stop a repeat.

Direct answer

I would build a small weekly habit of looking directly at real user behavior and words, using the cheapest channels, and I would track a decision log so I can see whether that evidence changes what the team builds. Closeness only counts if it moves a decision.

A week-to-week routine (about 75 minutes weekly, plus a 45-minute call every other week)

Check your company's privacy rules before watching recordings or joining calls.

How to tell whether it changes decisions

Keep a one-line decision log: decision, the user evidence behind it, and whether the evidence changed the plan. Each quarter, count the decisions where user evidence changed the scope, the order or the answer. If that count stays at zero for a quarter, the habit is a ritual.

Worked example (illustrative)

A data engineer owns a pipeline that feeds a dashboard. Ticket reading shows several users saying "the numbers are old at 9am". Shadowing a call shows managers opening the dashboard for a morning meeting before the refresh finishes. The team's plan was to make the job 20% faster. The evidence changed it: move the refresh earlier and show a visible "last updated" time. That one decision goes in the log as evidence-changed.

Pitfalls and cost

• Sampling bias: your sample leaves out some kinds of users, so it misleads you. Loud tickets are not typical, so sample randomly.
• Replays show what, not why. Pair them with tickets or a call.
• Privacy: mask personal data and follow policy.
• Time cost: under two hours a week on average, so protect it on the calendar or it vanishes.
• Ritual drift: reading without a log produces warm feelings and no change.

Direct answer

Pick an improvement where you can draw a line from your technical change to something a customer felt: faster pages, fewer failed payments, fewer crashes. Show the before-and-after measurement from the customer's side, not just internal numbers like CPU usage, and be honest about other factors.

Structure

1. Problem as customers experienced it. A complaint, a ticket tag or an abandonment point.
2. Change. One or two sentences, in plain terms.
3. Proof it reached the customer. Use a customer-side measure and a fair comparison:
   • A real-user measurement (timing taken from actual customer sessions, for example a small script that reads the browser's built-in performance timing when the checkout form becomes usable and sends it, with the device type, to analytics) rather than only a test machine.
   • A staged rollout (release to a small share of users first) so the old and new versions can be compared in the same week. The 10% who get the new code are the rollout group; the other 90% keep the old code and act as the control group, the comparison group that lives through the same weeks and events but does not get your change.
   • A customer signal such as tickets, abandonment or retries.
4. Caveats. What else changed in that period.

Worked example (illustrative numbers)

A frontend developer notices checkout feels slow on mid-range phones. They defer loading a large recommendations script until after the payment form appears. Rollout goes to 10% of users first. The metric is the 95th percentile (p95) page-ready time: the time within which 95 out of 100 visits finish, so it shows the slow tail that customers complain about, not the average.

In the 10% phase, p95 page-ready time was 3.9 s for the rollout group against 5.7 s for the control group in the same two weeks (illustrative), which is 1.8 s or about 32% faster. Customer-side signal: support tickets tagged "slow checkout" per 1,000 active users. Suppose that was 18 per 1,000 before and 11 per 1,000 after, comparing four-week windows. That is 7 fewer per 1,000, about a 39% reduction (7 divided by 18). The honest line to add: "a marketing campaign ended in the same window, so I also compared the 10% rollout group against the control group in the same weeks, and the gap held." The control comparison is the p95 gap above: the campaign reached both groups, so it cannot by itself explain a gap between them. This holds only if users were assigned to the rollout group at random (for example by a hash of the user ID), so say that in the story. Even then it supports 'contributed to', not 'proved': the ticket drop (18 to 11 per 1,000) is a before and after comparison that the campaign ending can still distort, so also report tickets per 1,000 for the rollout group against the control group over the same weeks.

Pitfalls

• Internal metric as proof. "Server time dropped" does not show a customer noticed.
• Averages hide the tail. Report the tail or the share of affected customers.
• No comparison group. Before-and-after alone cannot separate your change from the calendar.
• Overclaiming. Say "contributed to" unless you had a clean control.