Customer and User Obsession Questions

Direct answer

Accuracy is the share of answers the model gets right on a test set (a fixed collection of labeled examples we score it on). Happiness is what users feel when they try to get something done. When the first keeps rising and the second does not, I assume the metric is no longer pointing at the real problem. I would find what users actually experience, pick a metric that tracks it, and then show that moving it improves outcomes, ideally with an experiment.

How I decide what to optimize instead

1. Find where accuracy and experience part ways. Candidate causes: the errors that remain are the costly ones (a wrong answer on an important task counts the same as a trivial one), the gains went to cases users rarely meet, latency or wording hurts more than correctness, or the model is confident when wrong.
2. Slice the evidence. Break accuracy by user segment, task type and error severity, and compare with behaviour: task completion, retries and re-asks, abandonment, thumbs-down and support contacts.
3. Listen directly. Review a sample of real sessions where accuracy was right but users left unhappy, and talk to some users. This is where the cause is named, which no metric will do by itself.
4. Choose a candidate metric that reflects the lived cost, such as severity-weighted error or the rate of tasks completed without a retry.

How I would check I am right

• Before changing anything, write down the prediction: "improving X should raise task completion and reduce re-asks."
• Run a controlled test (A/B, where users are randomly split, some getting the new version and some the old) and look at the behaviour metric plus a guardrail (a metric that must not get worse) like latency and trust signals. Fix beforehand the smallest improvement worth acting on, for example re-asks falling from 14 to 11 per 100 sessions, and run until each group has enough sessions to separate that change from noise.
• Check offline too: does the new metric correlate with satisfaction ratings on past sessions? If not, drop it.

Worked example (illustrative)

A voice assistant's intent accuracy (the share of requests where it works out what the user wanted) rises from 91% to 94% on the test set. Users are no happier. Reviewing sessions shows most remaining errors are on a rare but costly action ("cancel my order" understood as "change my order"), while the improvements came in simple music requests. We switch to a severity-weighted error score (count each error by how costly it is, not equally) and measure the rate of re-asks (a user repeating or rephrasing the same request).

Numbers (illustrative): a 1,000-request test set has 900 simple requests and 100 costly actions. The team weights a simple error 1 and a costly error 10 (a judgment call, not something the data decides). Before: 60 simple errors and 30 costly errors, so 90 errors (91% accuracy), weighted 60 + 300 = 360 out of a worst case of 900 + 1,000 = 1,900, which is 18.9%. After the accuracy work: 30 simple and still 30 costly errors, so 60 errors (94% accuracy), weighted 30 + 300 = 330, which is 17.4%. Accuracy rose 3 points (errors fell from 90 to 60, a one-third cut in the error count) while weighted error improved only about 8% (360 to 330), because the 30 costly errors, only a third of the 90 errors before but 300 of the 360 weighted points, did not move. That matches users not feeling better. After the costly-confusion fix (costly errors 30 down to 10) accuracy is 96% and weighted error is 30 + 100 = 130, or 6.8%.

The fix itself is to resolve the costly confusions and add a confirmation step for destructive actions. In the live test, the check is re-asks per 100 sessions, say 14 for the old version and 9 for the new (illustrative). If re-asks fall and ratings rise in the test, the diagnosis held; if not, the metric was wrong too.

Pitfalls

Beware optimizing a proxy (a stand-in number that is easier to measure than what you care about) until it breaks. Do not average away segments, where one group may be getting worse. Do not declare success from a test set number alone.

Direct answer

A quick map of the data: dashboards are charts built from logs (records of what the system did, such as errors) and telemetry (usage data the product records automatically). I would treat the complaint as data about a gap in what we measure, not as an exaggeration. I would first ask the customer for specifics (who, which task, when), then look at that account's own numbers instead of the company-wide averages, and then fix the blind spot so one account's pain can no longer hide inside a healthy-looking aggregate.

Step 1: Get specifics from the customer (day 1)

• Ask for two or three concrete incidents: which user, which screen or task, what time, what they expected, what happened.
• Ask who is affected: one admin, one team, or everyone at the company. Ask for screenshots or a screen recording.
• Ask what "poor" means to them: slow, wrong results, errors, hard to use, or missing features. Each points to a different data source.

Step 2: Slice the data to this account

• Filter every dashboard to the account and its users: error rate, latency, task success, support contacts, logins, usage trend over the last 90 days.
• Compare the account with a baseline (what is normal for it or for similar accounts: same plan, size, region). A drop only this account shows is the evidence.
• Check whether the incidents they described appear in logs at all. If they do not, suspect instrumentation (the logging and metrics added to the product so it records what happens), not the customer.

Why averages hide this (worked example, illustrative numbers)

Suppose the account generates 2% of all requests and those requests are slow, while the other 98% are fast. The 95th percentile (P95: the response time that 95% of requests beat) ignores the slowest 5% of requests. The slow 2% all sit inside that top 5%, so for P95 to have 95% of all requests below it, the value must sit above 95 of the 98 fast requests per 100 (the slow 2 are all above it anyway). That is 95 / 98, about the 97th percentile of the fast traffic, a normal fast response time. The company-wide P95 barely moves (an aggregate, meaning one number summarising all requests, hides one account), yet every user at this customer has a bad day.

Other common blind spots:

• Events that fire only after the app loads, so failures before that (for example sign-in through the customer's company login system, called single sign-on or SSO) are never counted.
• Bot or internal-traffic filters that accidentally drop this customer's IP range.
• Errors the client swallows and retries silently, so the user waits but no error is logged.
• A workflow nobody instrumented, such as a bulk import or an admin screen.
• Metrics that count active users, which do not drop while a frustrated admin keeps logging in but stops using the feature.

Step 3: Close the loop with the customer

Tell them what we found, even if it is "we could not see it, and here is how we are fixing that". If a real defect exists, fix it and confirm it with them, not only with our dashboard.

What I would change so we catch it next time

1. An account-level health view for the top accounts by revenue (one table or page, a row per account, with columns such as error rate, p95 latency, weekly active users and open tickets, each beside last month's value), reviewed on a fixed cadence, for example weekly by the customer-success lead and monthly by product and engineering.
2. Alerts on per-account deviation from that account's own baseline, not only global thresholds.
3. Synthetic checks (scripted tests that run the key workflows on a schedule) for the workflows high-value customers rely on.
4. A link between support or customer-success notes and telemetry, so every "it feels slow" report gets a telemetry lookup and the result is recorded.
5. A periodic conversation with key accounts, because qualitative signal finds problems no metric was built to see.

Pitfalls

• Concluding "our data is fine, so the customer is wrong". Absence of evidence in an incomplete instrument is weak evidence.
• Building a bespoke fix for one account without checking whether similar accounts hit the same issue.
• Adding dozens of per-account alerts that nobody triages; start with the top accounts and a few workflows.

Direct answer

A strong answer shows a specific, observable way of noticing what others dismissed (usually a workaround or a pattern in data, not a hunch), how I checked it was real and sized it, how I made the case so someone acted, and what changed for customers. The story below is an illustrative skeleton; replace it with your own facts and measured outcomes.

Situation

I was a data analyst for a reporting product. Tickets about exports were closed as "answered" because the export worked, so nobody saw a problem.

How I spotted it

I read a sample of the ticket text rather than only the categories, and noticed customers saying things like "I download the report and rebuild the chart in a spreadsheet." That is a workaround: people doing by hand what the product should do. I then checked usage data. Sizing means turning a hunch into counts: of 9,000 weekly views of the sales report, about 3,400 (roughly 38%, illustrative) ended in an export, against about 5% for comparable pages.

How I checked it was real

• I called a handful of customers and asked what they did with the file. I treated their answers as explanation, not proof of scale.
• I counted distinct accounts (unique customer companies, so one account filing ten tickets counts once) using exports this way, not repeated tickets. Of 260 active accounts, 41 (about 16%) exported that report weekly, and 33 of the 41 were mid-size accounts, the segment (a group of similar customers, here by company size) we most wanted to keep.
• I read 60 export-related tickets from the quarter and found 22 describing the rebuild-in-a-spreadsheet workaround.

How I got it acted on

I wrote a one-page note: the customers' own words, how many accounts, the cost to them (time spent rebuilding weekly), and a small proposed fix. I took it to the product manager with the engineering lead present, framed around a goal the team already owned.

What changed

The team added the missing chart option to the report. Exports on that page fell from about 38% of views to about 14% over eight weeks, and tickets mentioning the rebuild workaround dropped from roughly 7 a month to 1 (all illustrative). In a real answer state the measured change and how long you watched.

Reflection

I would have brought support into the investigation earlier, because they already knew the pattern.

Pitfalls

Claiming credit for obvious problems, saying "I just knew", skipping how big it was, and omitting the person who acted on it. Credit the team that built the fix.

Direct answer

I would assume the feedback describes our most engaged customers, measure how far it differs from the whole customer base, and then both correct for it and go collect what is missing. Until then I would label findings as "what power users and large accounts say" and keep them away from decisions that depend on everyone else, such as onboarding or pricing.

Step 1: Judge representativeness against a known base

We know the customer base from our own records (plan, tenure, usage level, region, industry, account size). Compare the respondents on those same attributes.

• Respondent share vs base share for each segment, and the response rate per segment (responses divided by people invited).
• Whether volunteers are different: an interview volunteer is self-selected, so compare their usage (for example sessions per week) with the base's median.
• Whether the topic matters: power users are a good guide for advanced features and a poor guide for first-week experience.

Worked example (illustrative numbers)

The base has 10,000 users: 1,000 power users (10%), 3,000 regular (30%), 6,000 light (60%). The survey gets 500 responses: 300 power (60%), 150 regular (30%), 50 light (10%). Power users are over-represented six times, and light users are under-represented six times.

Share of respondents who say they like a new feature: power 80% (240 of 300), regular 50% (75 of 150), light 30% (15 of 50).

• Raw result: (240 + 75 + 15) / 500 = 66%.
• Reweighted to the base mix: 0.10 x 80% + 0.30 x 50% + 0.60 x 30% = 8 + 15 + 18 = 41%.

The headline falls from 66% to 41% just by weighting each group by its real size. Weighting is a correction, not a cure: the 50 light-user answers now carry 60% of the weight (they contribute 18 of the 41 points, about 44% of the weighted figure), so a small sample drives a large share of the headline, and the few light users who answered may still differ from the many who did not.

Step 2: What I do about it

1. Label and scope. State who the sample represents next to every finding.
2. Fill the gaps on purpose. Recruit the missing segments directly: targeted outreach to light and new users, short in-product questions, interviews with churned customers, and a small incentive.
3. Triangulate with behaviour. Usage data covers everyone, so check whether what respondents say matches what the full base does.
4. Weight the quantitative numbers (the counts and percentages, as opposed to quotes and stories) as above, and report the small-segment wobble.
5. Decide by decision type. Advanced-feature requests can lean on power users; changes that affect new or light users need their evidence first.

Pitfalls

• Treating a big response count as representativeness. 500 power users are still power users.
• Letting the highest-revenue account's request drive roadmap by default. Revenue weighting is a legitimate lens, but state it as such and show the account-count lens too.

Direct answer

When you never meet customers, obsession becomes a habit of asking "who is affected, and how will they experience this?" before choosing, and of using the closest available evidence about them: support tickets, error logs, product telemetry (usage data the product records automatically), and the people who do talk to customers. The test is behaviour, not belief. A skeptical colleague should be able to point at specific choices you made differently.

What a skeptical colleague would actually see

• A user-impact line in your design notes and reviews. Who is affected, what do they see when it fails, what do they do next? A real line looks like: "Impact: about 2% of checkouts (card payments over $500) hit this path; the user sees a spinner for 30 seconds, then a generic error, and may retry and be charged twice."
• Error and failure behaviour written for the person hit by it. The message says what happened, whether their data or money is safe, and what to do next.
• Metrics tied to a customer outcome. You alert on "checkouts failing" or "reports arriving late to users", not only on server load.
• Instrumentation chosen to answer customer questions. Instrumentation means the logging and metrics you add to code so you can see what it did. You log enough (with an identifier support can look up) that a ticket can be resolved without asking the customer to reproduce it.
• Regular proxy contact. A proxy is a stand-in source: someone or something that sees customers so you do not have to. You read a sample of recent tickets, sit in on a support or sales call now and then, and ask the people who do speak to customers what surprised them.

Worked example (a backend engineer)

A payment call times out. The default behaviour returns "Error 500". The obsessed version does three things differently:

1. The message tells the user whether they were charged and that retrying is safe or not.
2. The log line carries the order identifier, so support can answer "was I charged?" in one lookup.
3. The alert fires on the failed-checkout rate rather than on CPU, so the team learns about customer pain first.

Nothing here needed a customer conversation. It needed the habit of imagining the person on the other end and checking tickets to confirm.

The same habit in other seats

• Network engineer, during an incident: decide the order of fixes by which customers are cut off (for example the region carrying the most checkouts), and write the status update in terms of what customers cannot do.
• Mobile developer: treat battery drain and crashes as customer-facing features, and check crash rate on older phones, not just your own.
• Machine learning engineer: read a sample of real model outputs and complaints, not only the offline accuracy score.
• Data engineer: choose schemas and keys so support can trace one customer's records, and keep the fields analysts need to answer "who was affected".
• Data analyst: build the dashboard for the business intelligence (BI) reader who will use it, with the customer-outcome metric first and plain labels.

Trade-offs and pitfalls

• Proxies mislead. Tickets over-represent people who bother to write in, and telemetry shows what users did, not why. Arrange some direct exposure even if it is only one call a quarter.
• Do not claim to speak for customers from inside your head; cite the ticket, the log or the person who talked to them.
• If your work never changes because of customer evidence, you are doing the same job as before with a nicer label.