Structured Problem Solving and Decomposition Questions

Direct answer

I led an analysis of why trial-to-paid conversion had fallen. I decomposed it by acquisition channel, concluded one paid channel was low quality, and recommended cutting its budget. The recommendation was wrong because my branches were not independent: device type cut across channels and explained the gap. A teammate's question exposed it, I re-cut by device within channel, and we reversed the decision before the budget cut took full effect. The numbers below are simplified to show the mechanism.

Structured elaboration (situation, task, action, result, lesson)

• Situation: Trial-to-paid conversion dropped and leadership wanted to know where to cut or fix.
• Task: I owned the analysis and the recommendation.
• Action (the flawed part): I split the problem by channel because the data was easy to cut that way. I did not check whether channels differed in who they brought. I presented one cut, one explanation and one recommendation, with no written assumptions.
• How I found it: A product analyst asked what share of the paid channel's trials were on mobile web, where we had recently changed the sign-up flow. When I checked, the share was far higher than for other channels. I re-cut conversion by device within each channel.
• Result: Within each device the channels converted about the same. The channel gap was largely device mix. We paused the budget cut, fixed the mobile sign-up problem, and I added a standing review step.

Worked example of the flaw (illustrative numbers)

The overall gap is 26% minus 14%, twelve points, yet within each device the two channels convert identically. A split by channel alone shows a large effect that is entirely explained by who arrives on which device. The fault was the structure: branches that overlap in their causes are not independent explanations.

What I do differently now

• Check that the axis I split on is not correlated with another important variable before drawing conclusions.
• Write my assumptions at the top and ask a colleague to try to break the tree.
• Run a small, reversible pilot before an irreversible decision.
• State confidence in the recommendation, not only the recommendation.

Trade-offs and pitfalls in telling this story

• Choose a real failure with a real cause in the structure, not a data-bug excuse or a failure that was really someone else's.
• Own the decision. "The data was bad" is not an answer to "what went wrong in the structure".
• Do not skip how it was found. Interviewers want to see the detection habit as much as the mistake.
• A story with no lasting change in behaviour sounds like a story told for the interview.

Direct answer

Turn the mandate into arithmetic and questions before ideas. Work out what +20% means in extra daily users beyond the trend already under way, split DAU (daily active users: distinct users doing a qualifying action in a day) into new, retained and resurrected users, size each lever roughly, and see whether the pieces reach the target. Rank first hypotheses by users added, and test the cheapest in one-week experiments.

What to ask first

1. How is DAU defined and which action counts? If merely opening a notification counts, the easiest lever is cheap to game. This decides which metric to trust.
2. What is DAU today and what is it doing on its own? The target should be measured against that trend.
3. Why 20% and why six months? A board commitment is firmer than a guess, which decides whether the target is negotiable.
4. What can we not use? "No budget" shifts the plan from paid acquisition to retention and activation.
5. What must not get worse (opt-out rate, revenue, quality)? These are the guardrails (metrics that must hold steady while you push the main one).

Probe vague replies: "grow users" becomes "DAU or signups, for which segment, measured how, by when?"

The arithmetic (illustrative baseline of 500,000 DAU)

Target: 500,000 × 1.2 = 600,000. Compounded over six months that is 1.2^(1/6) = 1.031, about 3.1% a month. The trend is what DAU would do with no new work. If DAU already grows 1% a month, trend alone gives 500,000 × 1.01^6 ≈ 530,760. The plan must add about 69,000 users beyond trend, roughly +13% on top of the trend (1.2 / 1.0615 = 1.13), not +20%.

Decomposition and sizing

DAU on a day = users who return from yesterday + new users activated (reached first value, such as their first post) + lapsed users who resurrect. First-pass guesses, to be replaced by data. One way to size a lever (retention, illustrative inputs): 80,000 users sign up a month, and if day-30 retention rises 5 points, each monthly signup cohort (the group who joined in the same month) keeps 0.05 x 80,000 = 4,000 more daily users. Over the six months of the window that adds up to about 6 x 4,000 = 24,000 (the table rounds this up to 25,000, which flatters the lever by about 4%). Activation and resurrection are sized the same way: users affected x points of improvement you think is achievable x months of cohorts. Replace each input with a measured value as soon as it exists.

The 14,000 left over (69,000 minus 55,000; about 15,000 if the retention row is left at its unrounded 24,000) must come from a new surface (a new place users meet the product, such as a new app tab or channel) or segment, or from renegotiating the target. That is the finding to take to the executive.

One-week experiments, with thresholds set beforehand

Each experiment tests one row of the table: the first targets Activation, the second targets Retention. The examples below use a developer-facing product for the first and a content-recommendation product for the second, to show two different kinds of first-value moment; substitute your own product's.

• Developer onboarding: KPI (key performance indicator) is the share of new developers making a first successful API call within 24 hours. Rule (illustrative): ship if it rises at least 3 points and signup completion falls by less than 1.
• "Make recommendations more surprising": make it measurable as the share of recommended items from categories the user has not engaged with in 30 days, with click-through rate (the share of shown items that get clicked) held above an agreed floor. Rule (illustrative): carry it into a longer test if the share of recommendations from new categories rises from 8% to at least 12%, day-1 and day-7 return among exposed users are no lower than the control group's, and click-through rate stays within 0.3 points of a 5% baseline.

A one-week test reads leading indicators (early signals that predict the real outcome, such as the share of users who return on day 1 and day 7), not six-month DAU.

Pitfalls

Accepting +20% with no baseline trend, inflating DAU through notification opens that raise opt-outs, and treating a one-week lift as proof of a six-month outcome.

Direct answer

A good milestone sequence retires the largest unknowns first, cheapest first, and makes each milestone end in something you can evaluate, not just "work done". Tell one concrete story: what was uncertain at the start, how you ordered milestones against that uncertainty, one decision where the order mattered, and one split you would change with hindsight.

Structured elaboration

Use STAR (Situation, Task, Action, Result) with the Action as a sequence:

• Situation and Task: the project, the stakes, the constraint (time, data, team size).
• Action: list the milestones in order, each with what it proved, then why that order: usually data and labels exist and are trustworthy, then a simple baseline shows there is signal, then the model is integrated and run in shadow (scored but not acted on), then it takes partial real traffic with a human override, and only then optimized.
• Result: what each milestone told you and any decision it changed (including a stop or a pivot).
• Looking back: a specific different split, with the reason.

Worked example

Example story to adapt (qualitative, meaning described in words; add your real numbers only if verifiable). Project: routing incoming support tickets to the right team with a text classifier.

Situation and Task. Tickets were routed by hand; leadership wanted automation, with two engineers and a quarter.

Action.

1. Milestone 1: data and metric. Export past tickets with the team that finally resolved them, define "correct routing" with the support leads, build a held-out test set (tickets kept aside and never used for training, so the score reflects unseen cases). Proved whether labels were usable.
2. Milestone 2: baseline. Keyword rules, then a simple text model. Proved there was signal and set the baseline (the simple version any later model must beat).
3. Milestone 3: shadow mode. The model predicted on live tickets while humans routed as usual; we compared the two daily. Proved integration worked and exposed live-data differences.
4. Milestone 4: partial traffic. The model routed only high-confidence tickets (those where it was most sure of its choice), with an easy human override.
5. Milestone 5: improvement. A stronger model only for the categories error analysis (reading the mistakes to see which categories fail and why) showed were weak.

I put data and the metric first because it was the cheapest way to learn whether the project was viable at all, and left model sophistication last since it was the most expensive and least certain to matter.

What a go or no-go looked like (illustrative numbers). Milestone 2 bar agreed with the support leads: at least 75% correct on the held-out set. Keyword rules scored 55%, the text model 78%, so go. In shadow mode the model matched the humans on 80% of live tickets and the disagreements were read one by one; a pile of them were new product names, so a retraining step was added before partial traffic. Had the model sat at 60%, the call would have been no-go until labels or categories were fixed.

Result. Each stage ended in a go or no-go decision; shadow mode caught a mismatch between historic and live ticket wording before customers were affected.

What I would split differently. I defined labels in milestone 1 with one support lead; reviewers disagreed later about borderline categories. Now I would add a small labeling-agreement check to milestone 1 (two leads label the same 50 tickets independently and you count how often they pick the same team; 41 of 50 would be 82%, and a low figure means the categories need redefining before any model work), and bring the integration work forward so it overlaps with the baseline.

Trade-offs and pitfalls

• A calendar split ("month 1, month 2") is not a milestone split; each milestone must produce evidence.
• Starting with the most interesting model is the classic wrong turn.
• A story with no failed or changed milestone sounds rehearsed. Show a decision a milestone changed.
• "Nothing" as the answer to what you would split differently signals little reflection.
• Do not claim accuracy numbers you cannot defend; describe the decision each result supported.

Direct answer

Score every candidate on three things: how likely it is, how much of the problem it would explain if true, and how many hours it takes to check. Investigate in order of explained-per-hour, not by gut feeling or by whoever argues loudest. Cheap checks that can eliminate a cause go first, and each check gets a time limit.

The scoring rule

A hypothesis is a candidate explanation you can test. Priority = likelihood × share of the problem explained if true ÷ hours to check. The numbers are honest guesses, and their main value is that teammates can score separately and expose disagreement.

Worked example (illustrative numbers)

Checkout conversion fell from 3.0% to 2.7% a week ago. You have three days, about 18 working hours.

Likelihoods sum to 0.90, leaving 0.10 for "something else". To estimate "share explained if true", ask how many points of the 0.3-point drop (3.0% to 2.7%) the cause could produce if real, divided by 0.3. For B: if the campaign brings 10% of sessions and they convert at 1.2% instead of 3.0%, they pull the average down by 0.1 x 1.8 = 0.18 points, and 0.18 / 0.3 = 0.6. For A, a tracking bug would make the whole drop unreal, so 1.0. Order: A, B, C. Total effort is 12 hours, leaving 6 hours of the 18 for follow-up on whatever the checks reveal.

What would flip the order

If the release log shows nothing touched checkout tracking, A's likelihood falls to 0.05 and its score to 0.025. The order becomes B (0.060), C (0.027), A (0.025). A still costs only 2 hours, so it can run in parallel as a sanity check (a quick test that confirms a result is not an artefact).

Effort, risk and parallelism

• Independent checks run at the same time. With two analysts, A and B start together on day one, and the scores decide who gets a scarce third person.
• Add a cost-of-delay adjustment (weight a cause by what each extra day of waiting costs): if C means lost revenue every day and the fix is a one-line revert, move it up. For example, if a one-hour revert test replaces the six-hour investigation, C scores 0.20 x 0.8 / 1 = 0.160, above A's 0.150, so C runs first even though it is the least likely.
• Timebox each check (give it a fixed time limit), for example "if no signal after 4 hours, record that and move on." Allow one deep check only after the cheap ones fail.
• Write down what you ruled out and how, so the work is not repeated.

Pitfalls

Anchoring (locking onto the first idea or number you heard), treating the scores as measurements, and checking only evidence that confirms a favourite cause.

Direct answer

I would not pick one up front. I would do the cheapest step first, which is plotting the raw metric over time and slicing it by one or two obvious dimensions, because that costs minutes and tells me which heavier approach is worth running. Formal time-series modelling (fitting a model of what the metric normally does, then flagging departures) is the right lead when the series is noisy or seasonal enough that I cannot tell by eye whether it moved. Segment-first is the right lead when the metric is a blend of very different groups, because blends can move even when no group changed.

Structured elaboration

The two approaches answer different questions:

My decision order:

1. Check the measurement. Did a definition, a tracking event or a data feed change? Cheapest check, and it can end the investigation.
2. Plot the raw series with the previous periods overlaid. Is it a step on one date, a slow drift, or a repeat of last year's pattern?
3. Choose the next move from what you see. A sharp step points to what changed on that date, so slice by dimensions that changed then. A drift or ambiguity points to modelling the expected pattern. A metric that is a ratio or average points straight to segments, because its mix can shift.
4. Write the expected result before running each cut, so you know what would count as a finding.

Worked example

Conversion rate (orders divided by visits) falls from 3.5% to 2.75%.

Before: 70 / 2,000 = 3.5%. After: 55 / 2,000 = 2.75%. Neither segment's rate moved. The overall fell because the share of mobile visits rose from half to three quarters. A time-series model on the overall number would flag a real anomaly and still point at nothing fixable. The segment cut explains it in one step. The opposite case: a step down on one date in every segment equally points to a site-wide change on that date, and segments add little.

Trade-offs and pitfalls

• Going straight to modelling can burn days producing a precise answer to "when" when the real question is "where".
• Segment-first can mislead when the groups are too small. Check the group size before reading a difference.
• Segment by several dimensions one at a time before crossing them, and note how many cuts you made, since more cuts mean more chances of a chance difference.
• Overall, my default is raw plot, then segments, then modelling only if the picture stays unclear. I would flip to modelling first only if the data is highly seasonal with little history of mix change.