Customer and User Obsession Questions

Direct answer

I weigh the security benefit against the friction a customer will actually feel, using numbers where I can. Then I look for the lowest-friction way to get most of the protection, and I roll it out in stages with measurements and an exit. Security wins when the risk is serious and not otherwise reducible, but how it ships is a customer-experience decision.

How I weigh it

1. Risk removed. What attack or loss does the change prevent, how likely, and for whom? Ask security for a specific scenario, not "best practice."
2. Friction added. Which customers feel it, how often, at what moment (signing in daily versus a rare admin action), and what do they do when annoyed (abandon, call support, work around it)?
3. Cheaper routes to the same protection. Apply the strict control only where risk is high (risk-based or step-up checks: ask for an extra proof only when something looks unusual or the action is sensitive), remember trusted devices, or give a smoother factor.
4. Who carries the cost. Friction on a signup flow costs conversion; friction in an enterprise admin flow costs support calls.

Three related cases

Worked example (illustrative): rolling out MFA to 100,000 accounts

• Stage 1: 5% of accounts (5,000), chosen to include small and large customers. Track setup completion, lockout-related support contacts, and sign-in success.
• Gate: advance only if the thresholds agreed beforehand with security and support are met. Illustrative thresholds: at least 80% of prompted accounts finish setup (4,000 of the 5,000), no more than 10 lockout-related support contacts a week (2 per 1,000 accounts), and sign-in success falls by no more than 1 percentage point. Miss any one and I pause, fix the cause, and rerun the stage.
• Stage 2 and 3: widen, with in-app explanation of why, then move from encouraged to required on a published date. Keep a rollback and an exception path for customers with real blockers.

Pitfalls

Do not frame it as security versus customers. Do not roll out to everyone at once. Do not set a "will not hurt conversion" bar with no measurement behind it.

Direct answer

A strong story names the feedback, shows how I judged it was worth acting on (more than one independent source, tied to a goal, cheap to test), turns it into hypotheses I could check, ships in small steps, and reports what happened honestly, including what I would do differently. The version below is an illustrative skeleton to adapt with your own real facts and measured results.

Situation

I owned the setup flow of a team scheduling product. Support kept hearing "I finished setup but don't know how to add my team."

Deciding the feedback was worth acting on

• It came from independent sources: support tickets, notes from onboarding calls, and several observed sessions where people searched settings for an invite option.
• It touched a goal we already held: new accounts becoming active teams.
• It was cheap to test and low-risk if wrong.
• I set aside one request I did not act on: a single large account asking for a bespoke invite screen, because it was one voice and did not show up elsewhere.

Turning feedback into testable hypotheses

1. People cannot find the invite step because it lives under Settings.
2. Some people do not realize inviting teammates is the point of the product.

Action in incremental releases

• Release one: add an "invite your team" step at the end of setup, shown to a randomly chosen half of new accounts (an A/B test: new accounts are assigned at random to two groups that differ only in whether they see the change, so timing and the mix of customers are balanced between them on average, though random assignment does not guarantee matched groups and does not remove chance, so the groups must be large enough that a gap this size is unlikely to be luck, which is why I report the group sizes below) so I could compare.
• Release two (after reading results): add a short line explaining what teammates unlock, and a reminder in the product after the first day.
• I measured the share of new accounts that invited at least one teammate within their first week, plus support tickets on the topic.

Result

Over four weeks, about 1,000 accounts saw the new step and about 1,000 did not (illustrative figures). In the first week after signup, 43% of accounts that saw the step invited at least one teammate, versus 31% of those that did not: 12 percentage points higher, about 39% relative (12 / 31). Once the step went to everyone, tickets titled "how do I add my team" fell from about 40 a month to about 22. Replace these with the figures you measured, and state how long you watched.

Reflection

I would have run two or three user sessions before release one to catch the confusing wording sooner.

Pitfalls

Citing a result you cannot defend, claiming the change was your idea alone, and skipping how you decided the feedback was valid. Interviewers listen for judgement, not just activity.

Direct answer

I would default to the customer-visible onboarding change, unless the QA bottleneck is slowing everything the team ships. The decision turns on one comparison: how much customer value does each option create, on a common basis, for the same engineering cost? Internal tools are not worse in principle (faster QA means faster delivery of everything), but their benefit is indirect, so I would need evidence that the bottleneck is real.

Put both options on the same scale

• Customer-visible change: what share of new users fail to finish onboarding, and what would a fix plausibly move? Measure activation (reaching the first meaningful success, such as the first completed task).
• Internal tool: hours of QA time saved, delay between code complete and release, and bugs reaching customers.
• Convert both to "what would customers feel, and when?" For the tool, that is releases that arrive sooner or with fewer defects.

Evidence I would want

1. Onboarding funnel (the sequence of onboarding steps, with the share of users who drop off at each): where users leave, and what they say in support or session recordings (videos of real users' screens).
2. Cost of the fix: engineering weeks for each option, and how confident that estimate is.
3. QA data: how much time per release goes to the manual work, and whether QA is on the critical path of releases (the slowest chain of steps, so that if QA is slow, every release waits and can be held back until its checks pass).
4. A cheap test of the onboarding fix (a prototype shown to a few users) before committing.

Worked example (illustrative numbers)

Assume both options cost the same, about 6 engineer-weeks.

Onboarding: 10,000 signups a month, 40% finish onboarding, and the fix is hoped to raise that to 44%. That target is an assumption to test, but if it holds, 400 more activated users a month (4 points of 10,000).

QA tool: 3 QA engineers lose 6 hours each a week to the manual steps, so 18 hours a week, about 78 hours a month (18 x 4.33), roughly two working weeks of QA time. Customers feel it only if those hours would otherwise gate a release.

Putting both in one unit: dollars over the first 12 months. Illustrative assumptions: 25% of activated users become paying customers at $20 a month, and QA time is worth $50 an hour.

Onboarding: 400 activated x 25% = 100 new payers a month = $2,000 of new monthly revenue each month
  12 monthly cohorts, the first paying 12 months, the last 1 month: $2,000 x (12+11+...+1) = $2,000 x 78 = $156,000
  (ignores churn and build time)
QA tool:    78 hours x $50 = $3,900 a month x 12 = $46,800 (the ceiling, since it is only worth that if the freed time is put to use)

So onboarding is about three times larger, and the verdict holds unless fewer than about 7.5% of activated users pay ($46,800 / ($400 x $20 x 78) = 0.075). That payer rate is the number I would test first. One basis caveat: the onboarding figure is gross revenue while the QA figure is the value of saved labor, so they are not strictly the same kind of money. Taking an assumed 80% gross margin on the revenue, the break-even payer rate rises from 7.5% to about 9.4% ($46,800 / ($400 x $20 x 78 x 0.8) = 0.094), so the ranking survives but with a thinner margin than the headline three-times suggests. If the data showed releases waiting two days on manual QA every week, the tool would speed up every future customer change and I would flip.

Trade-offs and pitfalls

• The visible change risks being a guess about what users want. Mitigate with a quick test.
• Internal tools get chronically underfunded, so a rule such as a small standing share of capacity for team productivity prevents a permanent "later."
• Do not decide on whoever argues loudest.

Direct answer

Decide on severity and cost of delay (what each extra week of waiting costs), not on the 2% figure. If those customers carry a large share of revenue and the bug blocks work or loses data, waiting gets more expensive each day. I would interrupt the feature with the smallest team that can fix the bug, protect the feature date as far as possible, and tell product and sales what we are doing, why, and when.

Step 1: size it in business terms

Illustrative: 500 customers in total, with $30M of annual recurring revenue (ARR, the yearly subscription revenue) across all of them, so the average customer pays $60k. The 2% affected are the biggest accounts, averaging $200k each.

2% of 500 customers = 10 customers
10 x $200k = $2.0M affected
$2.0M / $30M total ARR = 6.7% of revenue
$30M - $2.0M = $28M across the other 490 customers = about $57k each

The affected revenue share (about 7%) is a more honest headline than "2% of customers": each affected account pays more than three times the average ($200k vs $60k).

Cost of delay, with illustrative numbers. Suppose the bug blocks their work and each of the 10 accounts has a 5% chance per month of leaving. Expected loss is 10 x 0.05 x $200k = $100k of ARR put at risk for each month the bug stays open (a yearly revenue figure, lost for good if a customer leaves). A feature expected to add $240k of ARR a year earns about $20k a month, so a one-month slip costs about $20k of revenue, delayed once. The units differ, which favours the bug even more: each month of delay risks $100k of yearly revenue permanently, against $20k of revenue pushed back once. Waiting on the bug costs more than slipping the feature, and if a small team can fix the bug in less than a month (an assumption to confirm with engineering), the feature slips by less than that.

Step 2: severity, trend, workaround, fix size

Is it data loss, blocked work or an annoyance? Is it growing as more accounts adopt the affected setup? Is there a workaround? Is there a contractual commitment (a promise written into the customer's contract, such as an uptime or data-safety guarantee)? How big is the fix?

Step 3: apply an interrupt rule you would have agreed before the argument

An interrupt rule is a pre-agreed statement of which kinds of bug stop planned work.

The table also needs to say what happens to combinations it does not list. A bug that blocks work but has a workaround is handled by fix size like the two 'slows work' rows, except that data loss interrupts regardless of any workaround. A bug that slows work with no workaround moves up to the interrupt row when the affected accounts are high-value or the slowdown is growing.

Step 4: protect the feature

Pull one or two engineers, not everyone. Re-estimate the feature date and state the new one. Cut feature scope before quality.

Explaining the call

• To product: affected revenue share, severity, trend, the cost of delay against the days the feature slips, and what we will not do.
• To sales: the account list, what to say to customers (acknowledged, workaround, fix date, named contact), and a request to flag contractual commitments. Do not promise a date beyond the estimate.

Variation: a minority on older devices versus a marketing banner

A minority of users on older phones suffer crashes or slowness, while marketing wants a heavy banner everywhere. Check the share of users on those devices (illustrative: 8% of 50,000 monthly active users is 4,000 people), whether they are active and valuable, and whether the banner is optional and reversible. A feature flag (a switch that turns code on for chosen users) can ship the banner to capable devices and a lightweight static version to older ones while the crash is fixed on schedule. A banner is optional and reversible; a crash is harm.

Pitfalls

• Letting the percentage frame the discussion.
• A silent slip of the feature date, which erodes trust faster than an announced one.
• What would flip my call: if the affected revenue is tiny and an easy workaround exists, the bug waits behind the feature with a firm date.

Direct answer

When you never meet customers, obsession becomes a habit of asking "who is affected, and how will they experience this?" before choosing, and of using the closest available evidence about them: support tickets, error logs, product telemetry (usage data the product records automatically), and the people who do talk to customers. The test is behaviour, not belief. A skeptical colleague should be able to point at specific choices you made differently.

What a skeptical colleague would actually see

• A user-impact line in your design notes and reviews. Who is affected, what do they see when it fails, what do they do next? A real line looks like: "Impact: about 2% of checkouts (card payments over $500) hit this path; the user sees a spinner for 30 seconds, then a generic error, and may retry and be charged twice."
• Error and failure behaviour written for the person hit by it. The message says what happened, whether their data or money is safe, and what to do next.
• Metrics tied to a customer outcome. You alert on "checkouts failing" or "reports arriving late to users", not only on server load.
• Instrumentation chosen to answer customer questions. Instrumentation means the logging and metrics you add to code so you can see what it did. You log enough (with an identifier support can look up) that a ticket can be resolved without asking the customer to reproduce it.
• Regular proxy contact. A proxy is a stand-in source: someone or something that sees customers so you do not have to. You read a sample of recent tickets, sit in on a support or sales call now and then, and ask the people who do speak to customers what surprised them.

Worked example (a backend engineer)

A payment call times out. The default behaviour returns "Error 500". The obsessed version does three things differently:

1. The message tells the user whether they were charged and that retrying is safe or not.
2. The log line carries the order identifier, so support can answer "was I charged?" in one lookup.
3. The alert fires on the failed-checkout rate rather than on CPU, so the team learns about customer pain first.

Nothing here needed a customer conversation. It needed the habit of imagining the person on the other end and checking tickets to confirm.

The same habit in other seats

• Network engineer, during an incident: decide the order of fixes by which customers are cut off (for example the region carrying the most checkouts), and write the status update in terms of what customers cannot do.
• Mobile developer: treat battery drain and crashes as customer-facing features, and check crash rate on older phones, not just your own.
• Machine learning engineer: read a sample of real model outputs and complaints, not only the offline accuracy score.
• Data engineer: choose schemas and keys so support can trace one customer's records, and keep the fields analysts need to answer "who was affected".
• Data analyst: build the dashboard for the business intelligence (BI) reader who will use it, with the customer-outcome metric first and plain labels.

Trade-offs and pitfalls

• Proxies mislead. Tickets over-represent people who bother to write in, and telemetry shows what users did, not why. Arrange some direct exposure even if it is only one call a quarter.
• Do not claim to speak for customers from inside your head; cite the ticket, the log or the person who talked to them.
• If your work never changes because of customer evidence, you are doing the same job as before with a nicer label.