Customer and User Obsession Questions

Direct answer

I would not decide from the crash share alone. First I would find out who the 5% are and what dropping them costs them. Then I would compare options short of an abrupt cutoff: freeze their version, offer a lighter experience, or set a long, well-communicated sunset (a planned end of support) with help moving. My default is a frozen version (the last release that still works on their devices: no new features, but it keeps receiving security fixes, which is why the table shows a branch to maintain) plus a published runway (the stretch between announcement and end date, during which people can prepare), not a hard cutoff, because the affected customers did nothing wrong.

Step 1: read the numbers carefully

30% of crash reports from 5% of users means a much higher crash rate per user:

older devices:   30 / 5  = 6.0 reports per 1% of users
other devices:   70 / 95 = 0.74 reports per 1% of users
ratio: 6.0 / 0.74 = about 8 times

Each older-device user generates about eight times as many reports. That is real, but check it first: are a few devices crash-looping (the app crashes again every time it restarts), is one fixable bug behind most of it, and do crash reports equal lost sessions for those users?

Step 2: look from the affected customers' side

• Who are they: loyal long-time users, people in lower-income regions, or a business fleet that cannot upgrade? Is the 5% a share of revenue too?
• What happens to them: can they still use the product somewhere else, or are they locked out of something important to their lives or work?
• What do they lose in switching: data, history, a workflow, a new phone's cost?

Step 3: compare options

Recommendation

Fix the top crash causes first, since they may be few. Then freeze older versions where they are, stop adding features there, and announce an end date at least a couple of release cycles away (with monthly releases that is two months at minimum; for a loyal or hard-to-reach group, for example announce on 1 March and end on 1 September) with in-app notices and migration help. Review the affected segment's revenue and mission importance before the date.

What would change my mind

If the platform vendor stops shipping security updates and we cannot patch around that, protecting those users and everyone else may force an earlier end date. If the affected group is concentrated in a vulnerable or high-value segment, I would extend the runway.

Pitfalls

Do not let crash dashboards decide alone. Do not announce a date without telling users what to do. Do not assume 5% is small if it is your most loyal customers.

Direct answer

I would default to the customer-visible onboarding change, unless the QA bottleneck is slowing everything the team ships. The decision turns on one comparison: how much customer value does each option create, on a common basis, for the same engineering cost? Internal tools are not worse in principle (faster QA means faster delivery of everything), but their benefit is indirect, so I would need evidence that the bottleneck is real.

Put both options on the same scale

• Customer-visible change: what share of new users fail to finish onboarding, and what would a fix plausibly move? Measure activation (reaching the first meaningful success, such as the first completed task).
• Internal tool: hours of QA time saved, delay between code complete and release, and bugs reaching customers.
• Convert both to "what would customers feel, and when?" For the tool, that is releases that arrive sooner or with fewer defects.

Evidence I would want

1. Onboarding funnel (the sequence of onboarding steps, with the share of users who drop off at each): where users leave, and what they say in support or session recordings (videos of real users' screens).
2. Cost of the fix: engineering weeks for each option, and how confident that estimate is.
3. QA data: how much time per release goes to the manual work, and whether QA is on the critical path of releases (the slowest chain of steps, so that if QA is slow, every release waits and can be held back until its checks pass).
4. A cheap test of the onboarding fix (a prototype shown to a few users) before committing.

Worked example (illustrative numbers)

Assume both options cost the same, about 6 engineer-weeks.

Onboarding: 10,000 signups a month, 40% finish onboarding, and the fix is hoped to raise that to 44%. That target is an assumption to test, but if it holds, 400 more activated users a month (4 points of 10,000).

QA tool: 3 QA engineers lose 6 hours each a week to the manual steps, so 18 hours a week, about 78 hours a month (18 x 4.33), roughly two working weeks of QA time. Customers feel it only if those hours would otherwise gate a release.

Putting both in one unit: dollars over the first 12 months. Illustrative assumptions: 25% of activated users become paying customers at $20 a month, and QA time is worth $50 an hour.

Onboarding: 400 activated x 25% = 100 new payers a month = $2,000 of new monthly revenue each month
  12 monthly cohorts, the first paying 12 months, the last 1 month: $2,000 x (12+11+...+1) = $2,000 x 78 = $156,000
  (ignores churn and build time)
QA tool:    78 hours x $50 = $3,900 a month x 12 = $46,800 (the ceiling, since it is only worth that if the freed time is put to use)

So onboarding is about three times larger, and the verdict holds unless fewer than about 7.5% of activated users pay ($46,800 / ($400 x $20 x 78) = 0.075). That payer rate is the number I would test first. One basis caveat: the onboarding figure is gross revenue while the QA figure is the value of saved labor, so they are not strictly the same kind of money. Taking an assumed 80% gross margin on the revenue, the break-even payer rate rises from 7.5% to about 9.4% ($46,800 / ($400 x $20 x 78 x 0.8) = 0.094), so the ranking survives but with a thinner margin than the headline three-times suggests. If the data showed releases waiting two days on manual QA every week, the tool would speed up every future customer change and I would flip.

Trade-offs and pitfalls

• The visible change risks being a guess about what users want. Mitigate with a quick test.
• Internal tools get chronically underfunded, so a rule such as a small standing share of capacity for team productivity prevents a permanent "later."
• Do not decide on whoever argues loudest.

Direct answer

Habits come from what a team looks at every week, what it is rewarded for, and how close it sits to real users, not from a values poster. I would start by making customer reality visible in work the team already does, add a few small recurring rituals, and then change the incentives. I would avoid launching a campaign. The team would see me use the evidence in real decisions first.

Phase 1: Make customers visible (first month)

• Ask the team to name their top three internal metrics, then ask: "What customer outcome does each one stand in for?" Gaps become the first discussion.
• Everyone, including me, sits in on customer calls or support sessions; I start by listening alongside them.
• Add one customer evidence line to existing meetings, such as a short "who did this affect?" note in the planning template.

Phase 2: Rituals and cadence

Mentoring juniors

Pair a junior engineer with a more experienced one for a customer interview. Ask them to write user impact in every pull request (PR, a proposed code change) description: who is affected and how we would know it worked.

Incentives and performance criteria

Add customer outcomes to review criteria: fixed issues that customers felt, evidence used in decisions, quality of handoffs. Reward the engineer who found a customer problem, not only the one who shipped fastest. Avoid bonuses tied to one count (tickets closed), which teaches gaming.

Scaling beyond one team

Publish a short customer-signal digest, share the decision log template with neighboring teams, and let early adopters present what changed. Standardize the habit, not the slogan.

Avoiding burnout

Rotate who handles the heaviest customer exposure, time-box ticket duty, and treat a customer issue as a team responsibility instead of an individual's emergency.

Pitfalls

• A slogan with no changed incentives teaches cynicism.
• Measure both sides: leading signs (customer contacts held, decisions with evidence) and outcomes (repeat issues, satisfaction), so the ritual does not become box-ticking.

Direct answer

I would build a closed loop with five stages: capture every issue in one place with consistent tags, triage it against published criteria, give each decision an owner and a time commitment, connect decided items to the roadmap, and tell the customer what happened. The loop fails most often at the last two stages, so I would design those first.

The loop

1. Intake and tagging. Support and customer success record each issue in one shared system with required fields: product area, customer segment, severity (blocked, degraded, annoyed), the customer's own words, and the account. Free-text complaints with no tags cannot be counted.
2. Triage criteria. A short weekly review (product, support lead, an engineering representative) scores items by how many distinct customers are affected, how severe the impact is, how strategic the segment is, and whether a workaround exists. Publish the criteria so the field knows why something was not picked.
3. Owners and SLAs. An SLA here is a service-level agreement: a stated time within which something gets a response. Every triaged item gets a named owner and a decision deadline, for example triage within a week and a decision within a month (illustrative). An item with no owner is how backlogs become black holes.
4. Reaching the roadmap. Group related items into problems, not feature requests. Problems with enough weight become roadmap candidates with the evidence attached; small fixes go to a standing bug-fix capacity (a slice of each sprint, for example 10 to 15% of engineering time, reserved for small fixes) so they do not wait for a planning cycle.
5. Closing the loop with customers. When an item is decided, support tells the customer: shipped, planned, or declined with a reason. A "no" with an explanation keeps trust; silence loses it.

Worked example

Support logs three tickets, from three different accounts, saying "can't find the invoice download". Tagged as billing area, small-team segment, severity degraded, they cluster with two more raised through customer success from two other accounts, so five tickets from five distinct accounts (counted by account, not by ticket volume). At triage they are grouped as one problem affecting five accounts with a workaround; the owner is a product manager with a decision due within the month. It ships as a small fix. Support then emails the five accounts, and the ticket closes only when the customer is told.

Model-improvement angle

When the product contains a machine-learning model, tag tickets where the model was wrong. Reviewed and with personal data removed, these become test cases (fixed examples the model must get right before each release) and training or evaluation examples (labelled examples used to teach the model or to measure how often it is right), so a ticket leads to a measurable model fix as well as a reply.

Pitfalls and measures

Measure the share of tagged items decided within the commitment and the share of decided items communicated back. Beware tagging fatigue (keep fields few), counting volume instead of distinct customers, and a loop that only handles complaints from the biggest accounts.

Direct answer

I would advise drawing the line around a small, named quality floor that the team does not trade for speed (things that destroy customer trust when they fail), and treat everything above the floor as scope that can flex to hit the market window. The trade-off becomes "cut scope, not the floor," and the evidence for the floor comes from trust signals, not opinion.

1. Define the floor from what breaks trust

Typical floor items: losing or corrupting customer data, security and privacy failures, wrong charges, and failures in the core flow that customers pay for. Everything else (polish, secondary features, edge-case flows) is above the floor.

2. Use trust signals with thresholds agreed in advance

Pick a few signals: support contacts per 1,000 active users, refund requests, failure rate in the core flow, and the share of new customers who return in week two. Agree beforehand: "If core-flow failures exceed the threshold, new feature work pauses until they are back under." A sample threshold (illustrative, each team sets its own): more than 2 failed core-flow attempts per 1,000 triggers the pause. Setting the number before looking prevents arguing it after.

3. Make the trade-off explicit and time-boxed

If a shortcut is taken above the floor, record it, give it an owner and a payback date. This is technical debt: a shortcut that works today but must be repaid later with rework. Unrecorded shortcuts become permanent.

Worked example (illustrative)

Six weeks to a market window. Four candidate items:

That is two items protected, one cut and one shipped partially. Advice to product: launch on time with a smaller surface, and publish the follow-up list.

What would change my call

If trust signals are already past the agreed threshold, I would advise pausing new launches, because speed on top of damaged trust compounds the loss. If the market window truly closes permanently (a regulatory date, say) and the floor items are intact, I would accept more shortcuts above the floor.

Pitfalls

• "Quality" as a vague word: name it, or the argument is taste versus taste.
• Engineering saying no without offering a smaller scope.
• No payback date for technical debt taken on.