Customer and User Obsession Questions

Direct answer

I would not agree yet. A rising Net Promoter Score (NPS) tells me that more survey respondents said they would recommend us. Before I treat that as "customers are happier" I want to know who answered, whether the rise is bigger than noise, and whether behaviour (retention, usage, support contact) moved the same way.

What NPS is

Customers answer one question, "How likely are you to recommend us to a friend or colleague?", on a 0 to 10 scale. Scores of 9 to 10 are promoters, 7 to 8 are passives, 0 to 6 are detractors.

NPS = % promoters - % detractors

The result runs from -100 to +100. Passives do not count in the formula.

What I would ask before agreeing

1. How many responded, out of how many invited, and did the response rate (answers divided by invitations) change? A different mix of respondents can move the score on its own.
2. Did the customer mix change (more enterprise, fewer trial users), or the survey wording, timing or channel?
3. How big is the rise relative to the survey's noise? See the example below.
4. What do the open-text "why" answers say? Did detractors shrink, or did promoters grow?
5. Did retention, renewals, usage depth and time to first value (how long new customers take to get their first real outcome) move in the same direction?

Worked example (illustrative responses, reproducible)

The code below measures noise, meaning the random wobble you get because a different set of customers answering would give a slightly different score even if nobody's feelings changed. To measure it, give every answer a value: +1 for a promoter, -1 for a detractor, 0 for a passive. NPS is 100 times the average of those values, which is exactly the share of promoters minus the share of detractors. The spread of the individual values is p + d - (p - d)^2, where p is the promoter share and d the detractor share. Dividing by the number of answers n and taking the square root gives the standard error: the typical amount the score would move if you re-surveyed a different random group of the same size.

standard error = 100 * sqrt( (p + d - (p - d)^2) / n )

Small surveys wobble more, because n sits under the square root. Two independent surveys combine their wobble as sqrt(se1^2 + se2^2), which is why a comparison between two quarters is noisier than either score alone.

Last quarter: 80 promoters, 70 passives, 50 detractors (200 answers), NPS +15. This quarter: 54, 36, 30 (120 answers), NPS +20. The VP sees +5.

from math import sqrt

def nps(promoters, passives, detractors):
    n = promoters + passives + detractors
    p, d = promoters / n, detractors / n
    score = 100 * (p - d)
    se = 100 * sqrt((p + d - (p - d) ** 2) / n)  # standard error of the score
    return score, se

q1, se1 = nps(80, 70, 50)
q2, se2 = nps(54, 36, 30)
diff = q2 - q1
se_diff = sqrt(se1 ** 2 + se2 ** 2)
print(round(q1), round(se1, 1))
print(round(q2), round(se2, 1))
print(round(diff), round(se_diff, 1), round(diff / se_diff, 2))

It prints 15 5.6, then 20 7.4, then 5 9.3 0.54. The standard error of the difference is about 9 points, so a 5-point rise is about half a standard error (5 / 9.3 = 0.54). A common rule of thumb is that a gap needs to be roughly two standard errors (here about 19 points, since 2 x 9.3 = 18.6) before you treat it as more than chance. With these samples a rise of 5 points cannot be told apart from chance, and the response count fell from 200 to 120, which itself needs explaining.

When another measure tells you more

Timing matters: CSAT and CES measure a moment, NPS measures a feeling about the relationship, so asking NPS right after a good support reply inflates it.

Limits of survey scores alone

Only responders are heard, scores can be gamed (asking only happy customers), and a number gives no reason. Pair every score with open-text answers and with behaviour. If NPS rises while renewals and usage are flat, believe the behaviour.

What I would tell the VP

"The rise is encouraging but not yet proof. Let me show you the segment split, the response counts and the retention trend next to it."

Direct answer

I would not present it as user experience against revenue. I would frame it as a bet about long-term value, state the harm as a testable hypothesis with guardrails, and look first for a version that captures the revenue without the harm. If there is no such version, I would propose an experiment with a holdout group (users who do not get the change) so the business decides on evidence.

Elaboration

1. Restate the business goal and agree what success means, for example revenue per new user over 90 days, not first-week revenue.
2. State the customer harm as a hypothesis: "if we show the upgrade prompt before a user has had a first success, fewer reach that success and 60-day retention falls".
3. Use counterfactual lifetime value. Lifetime value (LTV) is the total revenue a customer is expected to bring over their time with you. Counterfactual means comparing against what would have happened without the change, not against zero.
4. Set guardrail metrics: measures that must not get worse while you chase the target, such as week-4 activation (the share of new users who have completed their first meaningful task within four weeks of signing up), downgrades and support tickets. Each needs a threshold agreed in advance (a guardrail threshold), for example "stop the test if week-4 activation in the test group falls more than 2 points below the holdout".
5. Offer options that respect the goal: same prompt later in the journey, a softer prompt, or a smaller test.
6. One-page narrative for executives: the goal, the proposal, the risk in one sentence with its number, the experiment design, and the decision wanted by a date.

Worked example (illustrative numbers; a simple model that ignores discounting, which is the idea that money received later is worth less than money today)

Baseline: 10,000 trial users a month, 4.0% convert (400 payers), $20 per month, monthly churn 3%. Monthly churn is the share of payers who cancel each month. If 3% cancel each month, the average payer stays about 1 / 0.03 = 33.3 months (if 1 in 10 left each month, the typical stay would be 10 months). At $20 a month, LTV is about $667 per payer.

The proposed change lifts conversion by 20% (4.8%, 480 payers). My hypothesis is that it raises monthly churn to 4%, giving a lifetime of 25 months and an LTV of $500.

The change looks like a 20% win in month one and is about 10% worse over the lifetime. To break even at 4% churn, conversion must exceed 266,700 / 500 / 10,000, about 5.3%. The churn rise is my hypothesis, not a fact, and churn takes months to show. So the test uses leading indicators (early measures that move before the final outcome, such as week-4 activation) for an early read, and keeps a long-running holdout for churn. Illustrative design: each month 1,000 of the 10,000 trial users (10%) stay on the current flow and 9,000 get the change. Users are assigned to the two groups at random at sign-up, because random assignment balances the groups on known and unknown differences (traffic source, company size, intent), which is what lets the holdout stand in for what would have happened without the change. After four weeks compare week-4 activation between the two groups, and track each group's payer churn monthly. Activation is the early warning; 60-day retention is the later confirmation. A caution on the guardrail: with only 1,000 users in the holdout and 9,000 in the change group, a week-4 activation gap has a 95% margin of about 3.3 points (1.96 x square root of (0.25/1,000 + 0.25/9,000), taking activation near 50%), so a 2-point stop line is inside the noise (z of about 1.2) and a single month cannot trigger it reliably. Either pool two or three monthly cohorts before applying the 2-point rule, enlarge the holdout, or set the stop line at 4 points or more for a one-month read.

The holdout is the counterfactual, the comparison against what would have happened without the change. If the holdout's 60-day retention is 80% and the change group's is 74% (illustrative), the 6-point gap is the change's effect. Comparing 74% with last quarter's number instead would mix in seasonality and other launches.

Other versions of the same tension

• A UX issue that drives churn vs a monetization change: fixing the churn driver is itself a revenue decision. Size it with the same LTV arithmetic and compare it with the monetization idea.
• Short-term satisfaction vs long-term integrity: a design that tricks users into paying may satisfy the quarter and erode trust. Name it as a trust risk and offer an honest alternative with a clear value moment.

Trade-offs and pitfalls

• A loud "this will hurt users" without a number loses to a concrete revenue forecast.
• Do not hide behind the experiment to avoid taking a position. Say what you expect and why.
• Guardrails need thresholds agreed in advance, otherwise any result can be explained away.

Direct answer

I would assume the feedback describes our most engaged customers, measure how far it differs from the whole customer base, and then both correct for it and go collect what is missing. Until then I would label findings as "what power users and large accounts say" and keep them away from decisions that depend on everyone else, such as onboarding or pricing.

Step 1: Judge representativeness against a known base

We know the customer base from our own records (plan, tenure, usage level, region, industry, account size). Compare the respondents on those same attributes.

• Respondent share vs base share for each segment, and the response rate per segment (responses divided by people invited).
• Whether volunteers are different: an interview volunteer is self-selected, so compare their usage (for example sessions per week) with the base's median.
• Whether the topic matters: power users are a good guide for advanced features and a poor guide for first-week experience.

Worked example (illustrative numbers)

The base has 10,000 users: 1,000 power users (10%), 3,000 regular (30%), 6,000 light (60%). The survey gets 500 responses: 300 power (60%), 150 regular (30%), 50 light (10%). Power users are over-represented six times, and light users are under-represented six times.

Share of respondents who say they like a new feature: power 80% (240 of 300), regular 50% (75 of 150), light 30% (15 of 50).

• Raw result: (240 + 75 + 15) / 500 = 66%.
• Reweighted to the base mix: 0.10 x 80% + 0.30 x 50% + 0.60 x 30% = 8 + 15 + 18 = 41%.

The headline falls from 66% to 41% just by weighting each group by its real size. Weighting is a correction, not a cure: the 50 light-user answers now carry 60% of the weight (they contribute 18 of the 41 points, about 44% of the weighted figure), so a small sample drives a large share of the headline, and the few light users who answered may still differ from the many who did not.

Step 2: What I do about it

1. Label and scope. State who the sample represents next to every finding.
2. Fill the gaps on purpose. Recruit the missing segments directly: targeted outreach to light and new users, short in-product questions, interviews with churned customers, and a small incentive.
3. Triangulate with behaviour. Usage data covers everyone, so check whether what respondents say matches what the full base does.
4. Weight the quantitative numbers (the counts and percentages, as opposed to quotes and stories) as above, and report the small-segment wobble.
5. Decide by decision type. Advanced-feature requests can lean on power users; changes that affect new or light users need their evidence first.

Pitfalls

• Treating a big response count as representativeness. 500 power users are still power users.
• Letting the highest-revenue account's request drive roadmap by default. Revenue weighting is a legitimate lens, but state it as such and show the account-count lens too.

Direct answer

I would build the program backward from the decisions it must change: a small set of sources, one shared tagging scheme across product lines and languages, a named owner for each theme (the person accountable for one recurring customer problem), and a quarterly planning rule that every top roadmap item cites customer evidence or states why not. I would roll it out in about 90 days, starting with one product line, and judge it by whether decisions changed, not by report volume. If I could do only one thing first, it would be to agree the decisions with leadership and the rule that top roadmap items cite customer evidence, then pilot the tagging scheme on a single product line.

1. Decide the decisions first

Agree with product leadership which decisions it feeds: the quarterly roadmap, bug-versus-feature priority and sunsetting (retiring a feature). If no decision uses a theme, drop the theme.

2. Sources (capture)

Each record carries: account ID, segment (SMB or enterprise), annual recurring revenue (ARR), product line, language, source and date. This is what lets you weight by revenue and still show SMB voices. Integrate with the customer relationship management (CRM) system (where sales and account teams log customer contacts) and the support desk, so teams tag feedback in tools they already use.

3. Multilingual and multi-product handling

Keep the original text, add a machine translation (automatic software translation), and have native speakers spot-check a sample of each language. Tag with one shared taxonomy (the agreed list of tags every feedback item is labelled with) across the three product lines, so cross-product themes (such as slow exports) surface once rather than three times. An excerpt (illustrative):

Area (level 1)      Theme (level 2)        Example tags (level 3)
Data and exports    Export reliability     export-fails, export-missing-rows
                    Export formats         export-xlsx, export-api
Performance         Speed                  slow-search, slow-load
Onboarding          Setup                  sso-setup, import-data
Access              Sign-in                sso-missing, mfa-reset
On every item also: product line, language, severity (1 to 3)

Rules: tags are always in English whatever the original language, an item gets at most two tags, and new tags are added only at weekly triage so the list does not sprawl.

4. Triage and ownership

A rotating weekly triage (product manager, support lead, engineer) tags new items, links duplicates into one theme record (a single entry grouping all feedback about the same problem), and assigns a theme owner per product line. Propose a triage time target (for example, new items tagged within two working days) as a service-level agreement (SLA, a promised response time) the group commits to.

5. Quantify impact, then convert to roadmap

For each theme record: number of distinct accounts, ARR affected, frequency and severity. Example (illustrative): "Bulk export fails: 37 tickets from 14 accounts, 9 of them enterprise, no workaround." Each quarter the theme owner turns the top themes into one-page opportunity briefs (problem, evidence, size, options and a recommendation). In planning, each roadmap item lists its evidence, and every top theme gets a recorded decision: build, backlog, or decline with a reason.

6. Close the loop

Notify affected customers and the support team when a theme ships or is declined. That visible result is what keeps people logging feedback.

7. Rollout in 90 days

• Days 1 to 30: audit current sources, agree the taxonomy, pilot on one product line with one planning cycle in view.
• Days 31 to 60: connect the CRM and support desk, name owners, start weekly triage.
• Days 61 to 90: use it in a real planning cycle, review what changed, then extend to the other two product lines.

Stakeholders: an executive sponsor, a program owner, one theme owner per product line, and a contact in Support, Sales and Customer Success each.

8. How I would know it works (program-health indicators)

• Share of top roadmap items that cite customer evidence.
• Median days from first signal to a recorded decision.
• Share of themes with a decision within the quarter.
• Loop-closure rate (the share of customers who logged feedback and were told what happened to it).
• Coverage: feedback share per segment and language compared to the customer base.

Trade-offs

Central taxonomy with federated ownership (one shared tag scheme, with each product line's owner applying it and owning its themes) beats a central team that owns everything (it becomes a bottleneck) or fully independent teams (themes never connect).

Show both an ARR-weighted and an account-count view so enterprise does not drown out the SMB majority. For the bulk-export theme (14 accounts, 9 enterprise and 5 SMB), the account-count view says 5 of 14 accounts, 36%, are SMB. The ARR view, at an illustrative $60,000 per enterprise account and $4,000 per SMB account, is 9 x 60,000 + 5 x 4,000 = $560,000, of which enterprise is 96%. Shown alone, ARR would hide that more than a third of the affected accounts are small ones. A new channel later (for example a partner forum) slots into the same sources table and the same tags, because every source writes the same record fields.

Direct answer

I would triage every item on four questions: how severe is the harm, how many distinct customers are affected, is there a workaround (a way for the customer to still get the job done), and how much strategic or revenue exposure is involved. Severe, broad, no-workaround items go to engineering now; real gaps go to the backlog (the ordered list of work not yet scheduled) with a count attached; questions and misunderstandings are handled by support and documentation with no product work.

Decision rules

Worked example: five items arriving on one morning

1. Support: six reports from five accounts that CSV export drops the last rows since Tuesday's release. Escalate now. Data integrity, regression, several accounts.
2. Sales: one prospect says missing single sign-on (SSO, logging in once through the company's identity system) blocks the deal. Backlog, with the request recorded against that opportunity, because it is one account so far.
3. Customer: "How do I add a teammate?" No product work. Send the help article. If it keeps recurring, that is a backlog signal about the invite flow.
4. The executive of a very large account wants a custom dashboard colour. Backlog or decline with a reply, not an expedite (jumping the queue ahead of planned work).
5. Support: a checkout error for one customer on an old browser. Support handles it and logs it; escalate only if more accounts hit it.

Process behind the rules

• Single record per issue, not per ticket. Link duplicate tickets to one issue so the account count is honest. A recurring problem (for example, the same issue from several distinct accounts within a couple of weeks) is automatically promoted to a cross-functional owner. The exact threshold is a team choice.
• Tracking from first report to resolution: New, Triaged, Decided, In progress, Shipped, Customers notified. Each linked ticket is updated when the issue moves.
• Triage ownership and SLAs (service-level agreements, promised response times): a rotating daily triage owner (product manager with a support lead and an on-call engineer, the engineer currently responsible for urgent production problems) with a same-day target for escalations and a weekly review for the backlog. These are proposed commitments the group should agree on.
• Fairness across customers: write the expedite criteria down and apply them to everyone. A big account earns a faster response and clear communication, but not a product change unless the criteria are met. Count expedites per sprint (a fixed work cycle, often two weeks) so they do not swallow planned work.

One issue's life (illustrative)

The CSV export problem from item 1: Wednesday 09:00, status New, six tickets linked into one issue. 09:30, Triaged: five accounts, a regression, no workaround, so Decided: fix now. Wednesday afternoon, In progress: an engineer finds the paging bug. Thursday, Shipped. Thursday evening, Customers notified: support emails the five accounts and closes the six tickets.

A backlog entry, by contrast, looks like this:

Issue:            SSO login not supported
Accounts:         1 (a prospect with a $90,000 annual deal), 0 current customers
Workaround:       none for the prospect; password login for everyone else
Cost of waiting:  the prospect decides in 6 weeks
Re-review when:   3 accounts have asked, or 2 weeks before the prospect's deadline
Status:           Triaged, Decided: backlog

Pitfalls

Letting the loudest or biggest requester jump the queue, escalating everything (which dulls the signal), and closing a ticket without telling the customer.