Structured Problem Solving and Decomposition Questions

Direct answer

Weekly active users (WAU) is the number of distinct users who did a qualifying action in a 7-day window. I would not pull every cut of data. I would write a short set of testable hypotheses, rank them by how much of the drop each could explain and how cheap each is to test, run one cheap query that eliminates several branches at once, chase the biggest surviving branch, and drop branches by pre-agreed rules.

Step 1: structure the explanations

WAU this week = users retained from last week + new users + resurrected users (inactive before, active now). That is an arithmetic identity (an equation true by definition, so no user can fall in two parts), so the branches are exclusive and exhaustive. Add one more branch for measurement, since a logging change can fake a drop. Be precise about what this does: new, retained and resurrected partition the change in WAU, while Measurement and External are cross-cutting lenses (they ask why a component moved, so a broken event on one platform would show up inside 'retained'). They are not extra exclusive parts, which is why the 'share of drop' figures in Step 3 are upper bounds that sum to 1.80 and are used only for ranking, never added up.

Illustrative: WAU fell from 50,000 to 45,000, a loss of 5,000 (10%).

Step 2: the first query

One query splits the 5,000 loss into new, retained and resurrected change. Illustrative result: new -500, retained -4,000, resurrected -500 (total -5,000). The retained branch holds 80% of the drop, so it goes first. This single query also tells me acquisition and resurrection are minor.

Step 3: prioritise and test

Rank by (share of the drop it could explain) x (likelihood) / (cost to test). Use rough values, not false precision. Applying it to the five branches (illustrative; share comes from the first query, likelihood is my 0 to 1 judgement, cost is 1 for a quick check up to 3 for a big study):

Order: retained users, then measurement (a cheap check: event counts per platform), then the rest.

A filled hypothesis node (the template I reuse):

Step 4: when to drop a branch

• Size: it can explain no more than an agreed share of the drop (for example 10% or less, which is 500 of the 5,000). Applied to the first query: new users (-500) and resurrected users (-500) each sit exactly at 10%, so both are parked, and retained (-4,000, 80%) stays open.
• Timing: the suspected change happened after the drop began.
• Contradiction: the segment that should be hit hardest is not.

Versus a data dump

A boil-the-ocean pull (trying to analyse every possible cut of the data) produces dozens of cuts with no stopping rule, so the lead learns nothing for days. The hypothesis approach gives a first answer after one query and states what evidence would change the plan.

Issue tree versus hypothesis tree

An issue tree lays out all possible causes neutrally. A hypothesis tree starts from specific claims and tests the likeliest first. I build the issue tree to prove I have not missed a branch, then work as a hypothesis tree.

Pitfalls

• Falling in love with the first hypothesis and testing only to confirm it.
• No pre-agreed drop rule, so branches never die.
• Treating a correlation in one segment as proof of cause.

Direct answer

For each branch, write down what you would see in the data if it were true and what you would see if it were false, then choose checks where the three branches predict different things. Sequence by this rule: check the data first (tracking bug), then the cheap breakdown by traffic source, then the page-change investigation. If tracking is broken, every later number is suspect, so it goes first.

Structured elaboration

What each branch predicts (the evidence that separates them):

The hypothesis tree has the drop in conversion rate (the share of visitors who buy) at the root and three branches. A fingerprint is the pattern in the data that would appear only if that branch were true, and a deploy is a code release going live.

Conversion rate fell
|-- Tracking bug: sales happen but are not recorded
|-- Landing-page change: the new page persuades fewer visitors
|-- Traffic quality: the visitor mix shifted toward low-converting channels

Sequence and reasoning:

1. Tracking first (about an hour). Compare the tracked conversion count to backend orders for the same days. Why first: it is cheapest and invalidates everything else if wrong.
2. Traffic mix second (one query). Conversion rate by channel (where a visitor came from, such as paid ads or organic search), before and after, plus each channel's share of sessions.
3. Landing page third. Align the drop date with the release log (the team's record of what shipped when), compare old-page and new-page visitors if a holdout exists (a group of visitors deliberately kept on the old page), and read behaviour metrics.

Worked example

Illustrative numbers for step 2. Before: 10,000 sessions, of which 2,000 paid at 1.0% (20 conversions) and 8,000 organic at 2.5% (200 conversions), so 220 conversions and 2.2% overall. After: 4,000 paid at 1.0% (40) and 6,000 organic at 2.5% (150), so 190 conversions and 1.9% overall. The conversion rate dropped from 2.2% to 1.9% while no channel got worse. A dashboard showing only the overall rate would point at the landing page, and the channel breakdown shows it is a mix shift. Had backend orders also fallen 220 to 190 in step 1, the tracking-bug branch would already be dead.

The other two checks in numbers (illustrative). Step 1: tracked conversions after the drop are 190 and backend orders for the same days are 189, so tracking is sound. A tracking bug would look like 190 tracked against 220 backend orders, 30 missing sales (30 / 220 is about 14%). Step 3: organic converts at 2.5% both before and after, so the page did not hurt visitors in that channel. If the page were to blame, organic would read lower after, for example 2.0% (120 of 6,000) instead of 2.5% (150 of 6,000).

Trade-offs and pitfalls

• Branches can overlap. A campaign change can alter traffic and also coincide with a page release. Use a holdout or channel-level comparison to separate them.
• Do not stop at the first branch that fits. A tracking issue can hide inside a real drop.
• Mix shift averages hide problems. Always look at rates within segments, not only the total.
• What would change my order: if a release shipped the same day as the drop, I would still check tracking first, because a release is the likeliest cause of a tracking break as well.

Direct answer

ARPU (average revenue per active user) is revenue divided by active users. A driver is a factor that moves the metric when it changes, and a metric tree is a diagram that splits a metric into its drivers. Before any data, I split it into the share of active users who pay and the revenue each payer brings, then split the payer revenue into price, plan mix and add-ons. Each driver tells me which team holds the lever.

The metric tree

$\text{ARPU} = \frac{\text{revenue}}{\text{active users}} = \frac{\text{paying users}}{\text{active users}} \times \frac{\text{revenue}}{\text{paying users}}$

Plain English: ARPU falls if fewer active users pay, if payers each bring less (called ARPPU, average revenue per paying user), or both. Users either pay or do not, so the first split is mutually exclusive and exhaustive.

The third row is not a third factor to multiply. It is the denominator the pay share is measured against, listed separately because it can move on its own. The denominator is a driver too: if marketing brings in many free users, ARPU drops with no change in any paying customer's behaviour.

Worked example (illustrative numbers)

Before: 100,000 active users, 8,000 paying (8%), $25 per payer, so revenue is $200,000 and ARPU is $2.00. Now: 120,000 active users, 8,640 paying (7.2%), $25 per payer, so revenue is $216,000 and ARPU is $1.80, a 10% decline.

Revenue grew yet ARPU fell. The tree localises it: ARPPU is flat at $25, while pay share dropped from 8% to 7.2% (0.072 / 0.08 = 0.9, which is the 10%). So the first hypothesis is that the extra 20,000 users are mostly non-payers, pointing at the pay-share branch and the acquisition mix. That is a hypothesis for the data to confirm, not a finding.

Assumptions to write down

Define "active" (any login, or a qualifying action); revenue basis (billed means invoiced, collected means cash actually received, recognised means counted in accounting over the period the service is delivered); refunds in or out; one currency or converted; same time window for numerator and denominator.

Same method on two neighbouring metrics

• DAU (daily active users): new users + returning users + resurrected users (back after a gap). DAU also equals MAU (monthly active users) x stickiness, where stickiness is the share of monthly users active on a given day. For example, with 600,000 monthly users and stickiness of 20%, DAU is 120,000 (illustrative). Diagnostics: which component moved, and by cohort (users who started in the same period).
• Monthly revenue: users x paid conversion x price x (1 - refund rate). Each factor is measurable, and the assumptions above (definitions, window) must be stated before comparing months.

Pitfalls

• Reporting ARPU without its denominator definition.
• Treating ARPU as a customer-health metric when it can fall from healthy growth in free users.
• Stopping at one level: "price" is too coarse until split into list price, discounting and mix.

Direct answer

Order-processing time is a chain of stages, so break the target by stage first (measured with timestamps), then by region. For each candidate change estimate hours saved, effort and risk. Take quick wins that cover the target on paper, start the deeper change in parallel without counting on it inside six weeks, and re-forecast at a midpoint checkpoint.

Clarify the target

Is "time" the median, the average, or the P90 (the time within which 90% of orders finish)? Does 20% apply to each of the three regions or to the volume-weighted total (all orders pooled, so a large region counts for more than a small one)? Assume the median per region and confirm.

Stage breakdown (illustrative medians, one region)

Terms: pick and pack means warehouse staff collect items from shelves (pick) and box them (pack), done in batches called waves; carrier handoff is giving the parcel to the delivery company.

Target: 20% of 40 is 8 hours, so 32. Caveat: medians do not add. The median of total order time is generally not the sum of the stage medians, so treat the 40 as a sum of typical stage times, use the table to locate where the hours sit, and re-baseline the 32-hour goal from the median of end-to-end order timestamps (or work in means, which do add) before committing. Build the same table per region, since the levers differ.

Candidate levers

Type says what kind of change it is: a runtime setting is a value changed in the live system without building new software (a cutoff, a routing rule), Operations changes how people or sites work, and Data and model changes the input signals or the machine-learning model. Effort is in person-weeks (one person working full time for one week).

Where the hours-saved figures come from (illustrative reasoning)

• A: most of the 10-hour fraud stage is orders waiting for a person to review them. If auto-approving low-risk orders lets the typical order clear in about 7 hours, the saving is 10 - 7 = 3.
• B: allocation takes 6 hours and today runs after the fraud check. Running them side by side could save up to 6, but some orders need the fraud result before stock can be reserved, so plan on half: 3. This also allows for overlap with A, since both shorten the same stretch.
• C: with 2 waves a day the gap between waves is 12 hours, and if orders arrive evenly an order waits on average half the gap, 6 hours. With 4 waves the gap is 6 hours and the average wait is 3, a saving of 3.
• D: fewer false positives means fewer orders sitting in manual review. The 2 hours is the least certain figure, which is one more reason to keep D out of the six-week commitment.

A + B + C saves 3 + 3 + 3 = 9 hours, taking 40 to 31 (22.5%) against the required 20%. The buffer is thin (1 hour). With two people for six weeks (12 person-weeks, illustrative), A + B + C costs 1 + 3 + 4 = 8 and fits, while D at 10 does not.

Effort and risk by layer (the fraud example)

A false-positive fix spans three layers with different risk. A runtime change (threshold, queue routing meaning which review queue an order goes to, parallel steps) is reversible the same day. A model change needs offline validation and a shadow run, where the new model scores live orders silently beside the old one. For example, over one week the old model holds 100 orders and the new one holds 80, with 70 in common; reviewers inspect the 30 that only the old model held and the 10 that only the new one holds before the new model is allowed to decide anything (illustrative). A data change (new features (input signals the model uses) such as customer history) comes before both. So order work from reversible to validated.

Sequencing

Week 1: measure stage times per region, ship A with its guard. Weeks 2-4: B, and C in the pilot region. Week 3 checkpoint: re-forecast. If A + B + C forecasts under 8 hours, pull D's scope forward or renegotiate the slowest region's target. Week 6: measure the end-to-end median over a full week.

Pitfalls

Hours saved upstream can reappear as waiting at the next queue if pick and pack is the constraint (the slowest step, which caps how many orders flow through), so measure end to end rather than summing stage estimates. Averaging across regions can hide one that has not moved.

Direct answer

I first pin down what "dropped 15%" means, then build the tree on the identity renewals = subscriptions due x renewal rate. Below that I split by what happened to each customer who did not renew: they chose to leave (product, pricing), they could not pay (billing), or nobody helped them (customer success, CS). I would test the billing branch first because it is the cheapest to split out and cleanly actionable, and run the pricing check right after it (or in parallel if a second analyst is free).

Clarify first (illustrative assumption)

Assume 1,000 subscriptions came due in each quarter, and renewals fell 15%, from 800 (80%) to 680 (68%). That is 0.80 x 0.85 = 0.68. Non-renewals rose from 200 to 320, so 120 extra customers left. If the base of subscriptions due changed (for example more annual contracts, which renew once a year so a batch can land in one quarter, or a cohort, meaning a group of customers who signed up in the same period, that was weak), that is its own first branch.

The tree

Why this structure. Branch 0 checks the base, whether the number due changed. Branches 1 to 3 classify what happened to each lost customer, so every non-renewal has one primary home among them. The one-home test means checking that any given lost customer fits exactly one leaf, not two and not none. The tie rule says what to do when two leaves both fit: use the customer's recorded primary reason, and when it is missing, the earliest event in the sequence (for example a failed payment before any cancellation request counts as billing). Causes do interact (a price rise delivered with no outreach is both pricing and CS), so I say the tree classifies the primary cause, and I test interactions afterwards. The tree also gives four functions one shared map, which is why a disagreement among them stops being a debate.

Which branch first, and how I test it

Score each branch on size, how confident I am it is large, and cost to test. Billing wins on cost: the payment system already records failures, and the test is a count.

• Split the lost customers into involuntary (the payment failed, the customer did not decide to leave) and voluntary (the customer chose not to renew). Illustrative: failed-payment losses went from 40 of 200 to 100 of 320, an extra 60. So billing explains 60 of the 120 extra losses, half the drop.
• Next, the remaining 60 extra voluntary losses. Compare renewal rates of customers who received a price increase with those who did not, within the same plan (so plan type does not distort the comparison). This is observational, meaning I am comparing groups that already differ in what happened to them, not randomised, meaning nobody was assigned a price change by chance, so other differences between the groups could explain the gap. I read it as evidence, not proof, and check that the two groups are otherwise similar (tenure, size).
• Then product and CS branches using usage in the 30 days before renewal and outreach logs.

What would change my call

If a price change shipped this quarter and the failed-payment split is small, pricing moves to first. If the base of subscriptions due changed, branch 0 comes first.

Pitfalls

• Letting each function defend its own branch without a shared definition of "renewal".
• Testing the most interesting branch, not the most informative one.
• Treating an unexplained remainder as noise, not as the next branch to open.