Customer and User Obsession Questions

Direct answer

I would not present it as user experience against revenue. I would frame it as a bet about long-term value, state the harm as a testable hypothesis with guardrails, and look first for a version that captures the revenue without the harm. If there is no such version, I would propose an experiment with a holdout group (users who do not get the change) so the business decides on evidence.

Elaboration

1. Restate the business goal and agree what success means, for example revenue per new user over 90 days, not first-week revenue.
2. State the customer harm as a hypothesis: "if we show the upgrade prompt before a user has had a first success, fewer reach that success and 60-day retention falls".
3. Use counterfactual lifetime value. Lifetime value (LTV) is the total revenue a customer is expected to bring over their time with you. Counterfactual means comparing against what would have happened without the change, not against zero.
4. Set guardrail metrics: measures that must not get worse while you chase the target, such as week-4 activation (the share of new users who have completed their first meaningful task within four weeks of signing up), downgrades and support tickets. Each needs a threshold agreed in advance (a guardrail threshold), for example "stop the test if week-4 activation in the test group falls more than 2 points below the holdout".
5. Offer options that respect the goal: same prompt later in the journey, a softer prompt, or a smaller test.
6. One-page narrative for executives: the goal, the proposal, the risk in one sentence with its number, the experiment design, and the decision wanted by a date.

Worked example (illustrative numbers; a simple model that ignores discounting, which is the idea that money received later is worth less than money today)

Baseline: 10,000 trial users a month, 4.0% convert (400 payers), $20 per month, monthly churn 3%. Monthly churn is the share of payers who cancel each month. If 3% cancel each month, the average payer stays about 1 / 0.03 = 33.3 months (if 1 in 10 left each month, the typical stay would be 10 months). At $20 a month, LTV is about $667 per payer.

The proposed change lifts conversion by 20% (4.8%, 480 payers). My hypothesis is that it raises monthly churn to 4%, giving a lifetime of 25 months and an LTV of $500.

The change looks like a 20% win in month one and is about 10% worse over the lifetime. To break even at 4% churn, conversion must exceed 266,700 / 500 / 10,000, about 5.3%. The churn rise is my hypothesis, not a fact, and churn takes months to show. So the test uses leading indicators (early measures that move before the final outcome, such as week-4 activation) for an early read, and keeps a long-running holdout for churn. Illustrative design: each month 1,000 of the 10,000 trial users (10%) stay on the current flow and 9,000 get the change. Users are assigned to the two groups at random at sign-up, because random assignment balances the groups on known and unknown differences (traffic source, company size, intent), which is what lets the holdout stand in for what would have happened without the change. After four weeks compare week-4 activation between the two groups, and track each group's payer churn monthly. Activation is the early warning; 60-day retention is the later confirmation. A caution on the guardrail: with only 1,000 users in the holdout and 9,000 in the change group, a week-4 activation gap has a 95% margin of about 3.3 points (1.96 x square root of (0.25/1,000 + 0.25/9,000), taking activation near 50%), so a 2-point stop line is inside the noise (z of about 1.2) and a single month cannot trigger it reliably. Either pool two or three monthly cohorts before applying the 2-point rule, enlarge the holdout, or set the stop line at 4 points or more for a one-month read.

The holdout is the counterfactual, the comparison against what would have happened without the change. If the holdout's 60-day retention is 80% and the change group's is 74% (illustrative), the 6-point gap is the change's effect. Comparing 74% with last quarter's number instead would mix in seasonality and other launches.

Other versions of the same tension

• A UX issue that drives churn vs a monetization change: fixing the churn driver is itself a revenue decision. Size it with the same LTV arithmetic and compare it with the monetization idea.
• Short-term satisfaction vs long-term integrity: a design that tricks users into paying may satisfy the quarter and erode trust. Name it as a trust risk and offer an honest alternative with a clear value moment.

Trade-offs and pitfalls

• A loud "this will hurt users" without a number loses to a concrete revenue forecast.
• Do not hide behind the experiment to avoid taking a position. Say what you expect and why.
• Guardrails need thresholds agreed in advance, otherwise any result can be explained away.

Direct answer

A strong answer shows a specific, observable way of noticing what others dismissed (usually a workaround or a pattern in data, not a hunch), how I checked it was real and sized it, how I made the case so someone acted, and what changed for customers. The story below is an illustrative skeleton; replace it with your own facts and measured outcomes.

Situation

I was a data analyst for a reporting product. Tickets about exports were closed as "answered" because the export worked, so nobody saw a problem.

How I spotted it

I read a sample of the ticket text rather than only the categories, and noticed customers saying things like "I download the report and rebuild the chart in a spreadsheet." That is a workaround: people doing by hand what the product should do. I then checked usage data. Sizing means turning a hunch into counts: of 9,000 weekly views of the sales report, about 3,400 (roughly 38%, illustrative) ended in an export, against about 5% for comparable pages.

How I checked it was real

• I called a handful of customers and asked what they did with the file. I treated their answers as explanation, not proof of scale.
• I counted distinct accounts (unique customer companies, so one account filing ten tickets counts once) using exports this way, not repeated tickets. Of 260 active accounts, 41 (about 16%) exported that report weekly, and 33 of the 41 were mid-size accounts, the segment (a group of similar customers, here by company size) we most wanted to keep.
• I read 60 export-related tickets from the quarter and found 22 describing the rebuild-in-a-spreadsheet workaround.

How I got it acted on

I wrote a one-page note: the customers' own words, how many accounts, the cost to them (time spent rebuilding weekly), and a small proposed fix. I took it to the product manager with the engineering lead present, framed around a goal the team already owned.

What changed

The team added the missing chart option to the report. Exports on that page fell from about 38% of views to about 14% over eight weeks, and tickets mentioning the rebuild workaround dropped from roughly 7 a month to 1 (all illustrative). In a real answer state the measured change and how long you watched.

Reflection

I would have brought support into the investigation earlier, because they already knew the pattern.

Pitfalls

Claiming credit for obvious problems, saying "I just knew", skipping how big it was, and omitting the person who acted on it. Credit the team that built the fix.

Direct answer

I would not pick a winner by default. I would first ask what the dashboard metric actually measures, and whether it could be true at the same time as the frustration. Most often they are both right about different things or different people. Then I would run a short check of the metric against outcomes, decide on a conditional rollout with an explicit stop rule, and fix the cause.

Step 1: Find how both could be true

Worked example (illustrative numbers)

Task completion overall rises from 71% to 72.2%, which looks like success. Existing users are 70% of volume and new users 30%.

• Before: existing 80%, new 50%: 0.7 x 80 + 0.3 x 50 = 71%.
• After: existing 86%, new 40%: 0.7 x 86 + 0.3 x 40 = 72.2%.

Existing users improved by 6 points and new users fell by 10, so the interviews with frustrated newcomers were right and the dashboard was too.

Why the total rose although newcomers got worse: the total is a weighted average, and existing users carry 70% of the weight. Their 6-point gain adds 0.7 x 6 = 4.2 points to the total, the newcomers' 10-point loss subtracts 0.3 x 10 = 3.0, and the net is +1.2, which is 72.2 - 71. The 70/30 mix did not change here; the average simply hides two opposite movements. (Simpson's paradox is the stronger case, where a trend in every group reverses when groups are combined because group sizes differ. This is a cousin of it, not an instance.) That is why I split by segment before trusting an average.

Related cases and how I read them

• Positive quotes but a drop in the funnel (the steps from starting to finishing, such as sign-up to first success): quotes come from those who made it through. Study where the others left.
• Clicks up but users say results are irrelevant: a click shows curiosity, not value. Use what happens next: saves, purchases, returns, "not interested" signals.
• Conversion (the share of users who complete the desired action) up but tickets doubled: convert tickets to a rate. If tickets went from 120 to 240 while active users went from 10,000 to 20,000, the rate is 12 per 1,000 users both times, which is no worse. If users stayed flat, the rate really doubled. Also look at refunds, cancellations and support cost, because a confusing flow can lift conversion short term.
• A good average with a segment complaining: the segment is the finding.

What I decide

Do not declare success. Keep the feature live, hold any wider expansion, add a counter-metric (a second metric watched to catch harm, such as day-14 retention, the share of adopters still active 14 days after first use, or tickets per 1,000 users), and agree a stop rule before looking at next week's numbers.

An example stop rule (illustrative): before launch new-user task completion was 50% and tickets ran 12 per 1,000 active users. After launch, new-user completion is already 40% (the worked example above), so a rule written as 'stop if it drops below 45%' would already be breached and would just say stop. The honest version is therefore anchored to the current reading: expansion stays on hold now. We resume expanding only when new-user completion is back to at least 45% and tickets are at or below 15 per 1,000 in two consecutive weekly readings after the targeted fix. If new-user completion falls below 38% or tickets rise above 18 per 1,000 in either of the next two weekly readings, we pull the feature for new users while we fix it. The thresholds must sit outside the normal week-to-week wobble, which I check against earlier weeks, and I review every Monday for four weeks. Ship a targeted fix for the struggling segment and re-measure.

Which signal wins: the one closer to the user's real outcome and less exposed to selection (a bias where the measured group differs from everyone, for example only those who stayed). Neither the dashboard nor the quote file earns trust on its own.

Direct answer

Ranking by requester revenue measures who has leverage, not who is hurt or what matters to the strategy. It is a useful input (revenue at risk is real) but a poor ranking rule. I would rank requests on how many customers are affected, how severe the problem is, how well it fits the strategy and how much it costs to build, and handle documented churn risk in a separate lane.

Critique of the revenue rule

1. Leverage, not need. Revenue shows who can threaten to leave, not who needs the capability most.
2. Segment skew. Leverage here means bargaining power. The roadmap drifts toward your largest accounts and under-serves the segment you want to grow.
3. One voice, one data point. A large account's request is one account, however large its invoice.
4. Not comparable. Dollars on one side and problems on the other cannot be traded without a common scale.
5. Easy to game. Whoever reports the request can inflate the account's importance (gaming a score means feeding it inputs that push a favoured item up).

What to rank on instead

• Reach: share of active customers affected.
• Severity: 1 to 3. Anchors: 1 = annoyance, 2 = slows a task but a workaround exists, 3 = blocks work or loses data.
• Strategic fit: 1 to 3 (does it move where the company is going). Anchors: 1 = unrelated to the stated strategy, 2 = supports it, 3 = directly advances a named top priority such as moving up-market.
• Effort: engineer-weeks.
• Revenue at risk: its own churn-risk lane (a separate queue for requests tied to a customer likely to cancel, so it does not get mixed into the score), requiring evidence (renewal date, stated blocker).

Reach is how many, revenue is how much value, severity is how bad, fit is how much it matters to the strategy. They can disagree, which is why none should stand in for the others.

Normalising unlike inputs

Put them on defined scales: percentages for reach, 1 to 3 scales with written anchors for severity and fit, one unit for effort. Revenue is not a term in the score. It appears as a band in each brief and decides whether the churn-risk lane applies: band 1 under $100k, band 2 $100k to $500k, band 3 over $500k, so one large number does not drown the rest.

Variants of the same request

Group requests by the underlying problem, not the wording. "Export to CSV", "Excel download" and "weekly emailed report" all describe getting data out regularly. Count distinct accounts once and keep the variants as solution ideas.

Worked example (illustrative, 400 active customers)

score = reach % x severity x fit / effort weeks

Ranked by revenue: R1, R2, R3. Ranked by score: R2, R3, R1. R1 goes to the churn-risk lane only if there is evidence (renewal date, written blocker). Suppose R1's account renews in 90 days, with a written statement that custom SSO (single sign-on: one company login for many tools) attribute mapping, meaning passing the customer's own user fields such as department into our product, is a blocker. R1 is band 3 ($900k at risk) and costs 6 engineer-weeks. R2 costs 4. With 10 weeks of capacity, both fit, so build both, R2 first. With only 6 weeks, leadership compares $900k of retained revenue against R2's benefit to 12 accounts, and should first ask whether a 2-week configuration workaround retains the account.

Packaging the evidence for product

A one-page brief per request: the problem in one sentence, distinct accounts and users with the denominator, segment, severity evidence with a quote, revenue and renewal dates, variants merged, an engineering effort estimate, and what is still unknown.

Pitfalls

• The score sorts; it does not decide. Weights are choices, so write them down.
• Scores that depend on self-reported inputs get gamed. Require evidence for each number.

Direct answer

I would build the program backward from the decisions it must change: a small set of sources, one shared tagging scheme across product lines and languages, a named owner for each theme (the person accountable for one recurring customer problem), and a quarterly planning rule that every top roadmap item cites customer evidence or states why not. I would roll it out in about 90 days, starting with one product line, and judge it by whether decisions changed, not by report volume. If I could do only one thing first, it would be to agree the decisions with leadership and the rule that top roadmap items cite customer evidence, then pilot the tagging scheme on a single product line.

1. Decide the decisions first

Agree with product leadership which decisions it feeds: the quarterly roadmap, bug-versus-feature priority and sunsetting (retiring a feature). If no decision uses a theme, drop the theme.

2. Sources (capture)

Each record carries: account ID, segment (SMB or enterprise), annual recurring revenue (ARR), product line, language, source and date. This is what lets you weight by revenue and still show SMB voices. Integrate with the customer relationship management (CRM) system (where sales and account teams log customer contacts) and the support desk, so teams tag feedback in tools they already use.

3. Multilingual and multi-product handling

Keep the original text, add a machine translation (automatic software translation), and have native speakers spot-check a sample of each language. Tag with one shared taxonomy (the agreed list of tags every feedback item is labelled with) across the three product lines, so cross-product themes (such as slow exports) surface once rather than three times. An excerpt (illustrative):

Area (level 1)      Theme (level 2)        Example tags (level 3)
Data and exports    Export reliability     export-fails, export-missing-rows
                    Export formats         export-xlsx, export-api
Performance         Speed                  slow-search, slow-load
Onboarding          Setup                  sso-setup, import-data
Access              Sign-in                sso-missing, mfa-reset
On every item also: product line, language, severity (1 to 3)

Rules: tags are always in English whatever the original language, an item gets at most two tags, and new tags are added only at weekly triage so the list does not sprawl.

4. Triage and ownership

A rotating weekly triage (product manager, support lead, engineer) tags new items, links duplicates into one theme record (a single entry grouping all feedback about the same problem), and assigns a theme owner per product line. Propose a triage time target (for example, new items tagged within two working days) as a service-level agreement (SLA, a promised response time) the group commits to.

5. Quantify impact, then convert to roadmap

For each theme record: number of distinct accounts, ARR affected, frequency and severity. Example (illustrative): "Bulk export fails: 37 tickets from 14 accounts, 9 of them enterprise, no workaround." Each quarter the theme owner turns the top themes into one-page opportunity briefs (problem, evidence, size, options and a recommendation). In planning, each roadmap item lists its evidence, and every top theme gets a recorded decision: build, backlog, or decline with a reason.

6. Close the loop

Notify affected customers and the support team when a theme ships or is declined. That visible result is what keeps people logging feedback.

7. Rollout in 90 days

• Days 1 to 30: audit current sources, agree the taxonomy, pilot on one product line with one planning cycle in view.
• Days 31 to 60: connect the CRM and support desk, name owners, start weekly triage.
• Days 61 to 90: use it in a real planning cycle, review what changed, then extend to the other two product lines.

Stakeholders: an executive sponsor, a program owner, one theme owner per product line, and a contact in Support, Sales and Customer Success each.

8. How I would know it works (program-health indicators)

• Share of top roadmap items that cite customer evidence.
• Median days from first signal to a recorded decision.
• Share of themes with a decision within the quarter.
• Loop-closure rate (the share of customers who logged feedback and were told what happened to it).
• Coverage: feedback share per segment and language compared to the customer base.

Trade-offs

Central taxonomy with federated ownership (one shared tag scheme, with each product line's owner applying it and owning its themes) beats a central team that owns everything (it becomes a bottleneck) or fully independent teams (themes never connect).

Show both an ARR-weighted and an account-count view so enterprise does not drown out the SMB majority. For the bulk-export theme (14 accounts, 9 enterprise and 5 SMB), the account-count view says 5 of 14 accounts, 36%, are SMB. The ARR view, at an illustrative $60,000 per enterprise account and $4,000 per SMB account, is 9 x 60,000 + 5 x 4,000 = $560,000, of which enterprise is 96%. Shown alone, ARR would hide that more than a third of the affected accounts are small ones. A new channel later (for example a partner forum) slots into the same sources table and the same tags, because every source writes the same record fields.