Growth, Activation and Retention Questions

Direct answer

Segment customers by what they are trying to get done and how far they have progressed, not by a single label such as company size. Then build the roadmap in two layers: a shared core that every segment needs, and a small number of segment bets, each tied to one adoption obstacle that is specific to that segment. A segment that adopts slowly for a fixable reason gets a product change. A segment that adopts slowly because the product is the wrong fit gets deprioritised openly.

How to segment (and what to check before trusting a cut)

Adoption means a customer reaching and repeating the behaviour that delivers the product's value (for example, a team running a weekly report, or a user completing a core task). Build segments from three families of attributes, then keep only the cuts that show different adoption behaviour:

Validation steps, in order:

1. Outcome first. Pick one adoption measure with a clear definition (for example "reached the core action 3 times within 28 days of signup"). Without it, segments cannot be compared.
2. Cohort the data. A cohort is a group of customers who signed up in the same period (for example, everyone who joined in March). Compare each segment's cohorts at the same age, such as adoption at day 28, so older customers who have had more time do not flatter a segment.
3. Test whether the cut matters. A segment is useful only if adoption rates differ meaningfully between segments and each segment is large enough to act on. Check by comparing adoption curves (the adoption rate plotted by days since signup, one line per segment: lines that sit far apart mean the cut matters). Then use a decision tree, a model that repeatedly splits customers on the attribute that best separates adopters from non-adopters, so the first splits show what matters (for example, "connected their data in the first 3 days" may separate 70% adopters from 10%).
4. Check stability. If a segment's membership flips every month (a customer is a "power user" in March and "casual" in April), it is a behaviour state, not a segment.
5. Add qualitative evidence. Ten to fifteen interviews per segment explain why the number is what it is.

Turning segments into a roadmap

For each segment, write down: size (customers and revenue), current adoption, the single biggest obstacle, and the cheapest intervention that could move it. Illustrative output:

Then sequence by expected value: segment size and strategic value, multiplied by the realistic adoption lift, divided by effort. Keep roughly two thirds of capacity on the shared core (reliability, onboarding basics, performance) because those lift every segment, and put the rest into at most two or three segment bets so the roadmap does not fragment.

Worked example

Suppose 10,000 customers: 6,000 small self-serve at 55% adoption, 3,000 mid-size at 30%, 1,000 enterprise at 15%. Adopters today: 6,000 x 0.55 + 3,000 x 0.30 + 1,000 x 0.15 = 3,300 + 900 + 150 = 4,350 (43.5% overall). A guided-setup change expected to lift small teams by 5 points yields 6,000 x 0.05 = 300 more adopters. An invite-flow change lifting mid-size by 8 points yields 3,000 x 0.08 = 240 more. Now add the enterprise fix, SSO and audit logs, expected to lift enterprise adoption by 10 points: 1,000 x 0.10 = 100 more adopters. Ranked by extra adopters: small teams 300, mid-size 240, enterprise 100, so enterprise is last.

Now weight by revenue. Illustrative annual revenue per adopting account: small 1,000 USD, mid-size 5,000 USD, enterprise 40,000 USD. Extra revenue: small 300 x 1,000 = 300,000 USD; mid-size 240 x 5,000 = 1,200,000 USD; enterprise 100 x 40,000 = 4,000,000 USD. Ranked by revenue, enterprise is first and small teams last.

Finally divide by effort, the third term of the expected-value rule. Illustrative effort in engineer-weeks: small 4, mid-size 6, enterprise 16. Revenue per engineer-week: small 300,000 / 4 = 75,000 USD; mid-size 1,200,000 / 6 = 200,000 USD; enterprise 4,000,000 / 16 = 250,000 USD. The two rankings disagree, so state both, then choose using strategy and confidence: enterprise leads on revenue per week but is the least certain lift and has the longest lead time, so a sensible plan funds the mid-size and enterprise bets and keeps small-team setup as a cheap quick win.

Trade-offs and pitfalls

• Too many segments produce a roadmap of many small, unshippable pieces. Cap the active bets.
• Averages hide the story. A flat overall adoption rate can hide one segment growing and another collapsing.
• Demographic-only cuts (industry, region) often describe who customers are, not how they behave, and rarely suggest a product change.
• Equal treatment is not equal investment. Treating segments differently means different obstacles get different fixes, not that each gets the same effort.
• Re-measure each quarter and retire a segment bet whose adoption lift did not show up.

Direct answer

I would test a day-2 nudge that shows new users listings or items matching what they browsed, against no nudge, randomised at the user level. The primary metric is week-1 retention: the share of new users who come back on any of days 2 to 7 and do a core action. Illustrative sizing (baseline 30%, wanting to detect +3 percentage points) is 3,760 users per arm. I would stop early only for harm, not for a promising p-value (the chance of seeing a gap this large if the nudge did nothing).

Hypothesis

A marketplace app loses new users because the first session did not show them something relevant to them, and nothing pulls them back. So: "Sending new users, 24 to 48 hours after first open, a push and email of listings similar to what they viewed will raise week-1 retention versus sending nothing, because it supplies the relevant inventory they did not find the first time." Write it down before launch with the expected direction and size.

Metric definitions

• Primary: week-1 retention = new users with at least one core action (view a listing detail, message a seller or buy) on a day from 2 to 7 after first open, divided by all users randomised. Using "any session" would count accidental taps that came from the nudge itself.
• Secondary: purchases per new user in week 1, and 28-day retention (does the lift persist, or did we just pull the visit forward by a few days?).
• Guardrails: push opt-out rate, uninstall rate, email unsubscribe and spam-complaint rate, and seller-side effects (response times, in case demand spikes in thin categories, meaning categories with few sellers).

Who is in the experiment

All new users in one set of launch markets who have notifications enabled or an email on file, randomised 50/50 at first open. Excluded: staff and test accounts, users who joined through a partner flow with its own messaging. Randomise by user id using a stable hash (a fixed scrambling function of the id, so the same user always lands in the same arm). Use intention to treat: analyse everyone assigned, even if their push was not delivered; otherwise you select on a post-assignment variable.

Size, with the arithmetic

Assume a 30% baseline (illustrative; use your own), a minimum detectable effect (MDE, the smallest lift worth detecting) of 3 percentage points, a 5% two-sided significance level (a 1-in-20 chance of calling a win when the nudge does nothing, counting a drop as well as a rise) and 80% power (an 80% chance of detecting a true lift of 3 points):

Here z_{1-alpha/2} = 1.96 and z_power = 0.8416 are cut-offs on the normal curve: 1.96 leaves 2.5% in each tail, 0.8416 leaves 20% in one tail. Substituting p1 = 0.30 and p2 = 0.33: (1.96 + 0.8416)^2 = 7.849; p1(1-p1) + p2(1-p2) = 0.21 + 0.2211 = 0.4311; (p1-p2)^2 = 0.0009; so n = 7.849 x 0.4311 / 0.0009 = 3,759.5, rounded up to 3,760. The code does the same; inv_cdf converts a probability into its z cut-off.

from statistics import NormalDist
from math import ceil

def n_per_arm(p1, p2, alpha=0.05, power=0.80):
    z = NormalDist().inv_cdf
    return ceil((z(1 - alpha / 2) + z(power)) ** 2
                * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2)

print(n_per_arm(0.30, 0.33))

This prints 3760: 3,760 users per arm, 7,520 total. If you need to detect only +2 points (p2 = 0.32) the same substitution gives 7.849 x 0.4276 / 0.0004 = 8,390.4, so 8,391 per arm, because the required sample grows with the inverse square of the effect. Check feasibility: if 1,000 new users arrive per day, enrolment takes about 8 days; add 7 days of follow-up for the last enrolled user. Enrol across at least one full week so weekday and weekend behaviour is balanced.

What would make me stop early

Stop only for pre-agreed reasons:

• Harm: unsubscribe, complaint or uninstall rate in the nudge arm exceeds control by a pre-set margin (agree it up front, for example a rise large enough to cost more than the expected retention gain).
• Broken experiment: sample ratio mismatch (the arms are not near 50/50, which signals an assignment or logging bug), or a delivery failure in one arm.
• Not for a lead: peeking at the retention p-value daily and stopping at the first dip below 0.05 inflates false positives. If early stopping for a win is wanted, choose a sequential testing method in advance (a procedure that raises the bar at each interim look so repeated peeking does not inflate false positives) and size for it.

Decision

Ship if the primary metric lifts by at least the MDE, the 28-day retention is not worse and no guardrail tripped. A 1-point lift that is statistically significant but below the MDE is a "iterate", not a ship.

Pitfalls

• Measuring opens or push clicks (activity the nudge creates) instead of retention.
• Interference: in a marketplace, extra buyers in one arm can drain the same sellers' inventory seen by the other arm, shrinking the measured effect. A small share of traffic limits this.
• Running only on one platform and generalising to the other.

Direct answer

Seven segments below, each with a hypothesis and one success metric. The method: choose segments by behaviour first (it predicts value and is actionable), add demographic or geographic cuts only where the hypothesis needs them, and rank them by size x expected lift x ease before testing. Every experiment uses a randomised holdout within the segment (a random share of that segment who do not get the change, kept as the comparison group).

Segments, hypotheses, and metrics (consumer app, illustrative)

Count check: seven segments, each with one hypothesis and one metric.

How to prioritise

Score each: (segment size) x (plausible lift) x (value per converted user) / effort. Illustrative arithmetic with 100,000 new users: segment 1 is 40% of them (40,000 users) with a plausible lift of 3 points, so 40,000 x 0.03 = 1,200 extra converted users, each worth 20 USD of first-year gross profit, or 24,000 USD. Segment 3 is 2% (2,000 users) with a lift of 5 points: 2,000 x 0.05 = 100 extra users, each worth 200 USD (a power user who upgrades), or 20,000 USD. In users, segment 1 wins 1,200 to 100; in value they are close (24,000 against 20,000). Now divide by effort: if segment 1 takes 6 engineer-weeks and segment 3 takes 3, value per week is 4,000 against 6,667, so segment 3 ranks first. Compare in expected paying or retained users and their value, not in percentage points. Run the top two or three first, and use low-effort tests to learn about the others.

Experiment hygiene

• Randomise within a segment into treatment and control; do not compare segment to segment.
• Fix the metric, the minimum detectable effect (the smallest lift the test is sized to detect reliably, for example 2 points) and the duration in advance; run at least one full weekly cycle.
• Watch for overlap: a user in segments 1 and 4 should be assigned to only one test at a time.
• Guardrails (metrics that must not get worse while the target metric improves): notification opt-out rate, uninstall rate, crash rate.

Trade-offs and pitfalls

• Demographic targeting can be ethically and legally sensitive (minors, protected attributes, regional privacy rules); use behaviour where it works equally well, and take advice from legal and privacy teams before targeting by age.
• Small segments produce noisy results; if a test would take six months to reach significance (a result clear enough that chance alone is an unlikely explanation), pool segments or choose a larger effect. Pooling means testing two similar small segments together as one group to get enough users, at the price of a vaguer hypothesis, because the measured lift is an average over both.
• Do not overfit: a segment that exists only in this quarter's data is not worth a standing programme.

Direct answer

A segment that starts with high ARPU (average revenue per user per month), then neither expands nor stays past month six, usually has one of three problems: it was sold more than it can use (the early spend is not tied to realised value), it hits a value wall after initial setup, or it lacks a reason to deepen use (no expansion path). My three-part plan: (1) diagnose where and why value stalls, using survival and usage data, (2) change the product's first six months so value is reached and extended before month six, (3) add a path for the segment to grow into higher usage. At 90 days I would judge leading indicators on customers approaching month six, and at 180 days the actual survival and expansion past month six against a holdout.

Part 1: Analytic diagnosis (weeks 1 to 4)

1. Survival curve and hazard by cohort. A survival curve plots the share of customers still active by month; the hazard for a month is the probability of leaving in that month among those still active at its start: (active at start - active at end) / active at start. Illustrative cohort of 1,000: active at the end of months 0 to 7 are 1,000, 960, 925, 895, 870, 850, 610, 585. The hazard is 4.0% in month 1, falling to 2.3% in month 5, then (850 - 610) / 850 = 28.2% in month 6, then 585 vs 610 gives 4.1% in month 7. A spike like month six points to a contract end, a promotion expiring, or a value cliff; a gradual decline points to disengagement.
2. Compare stayers and leavers on early behaviour. Features used in months 1 to 3, seats activated (seats are the licences a customer buys, one per user; an active seat is one actually used), integration count, support tickets. Find which behaviours by month three separate those who stay past month six.
3. Compare price paid against usage. If high initial ARPU came from a large first purchase (many seats, big commitment, a promotional bundle) while usage per paid unit is low, the customer is over-bought and will downsize (contraction: staying a customer but paying less, for example by dropping seats) at renewal.
4. Talk to ten leavers and ten stayers. The data show what; interviews show why.

Output: a ranked hypothesis list with evidence for each.

Part 2: Product plan for months 1 to 6

Pick the interventions the diagnosis supports, for example:

• Value milestones: define two or three milestones that predict staying (such as a second team onboarded or a core workflow running weekly) and guide customers to them by month three.
• Early-warning health score: a health score (a single number summarising how likely an account is to stay) built from usage trend, seats active versus paid, and feature breadth, so in-product prompts and the account owner act at month four, before the exit decision.
• Right-sizing offer: let over-bought customers (those paying for more seats or capacity than they use) move to a better-fit plan before month six rather than leave; this lowers ARPU a little but protects lifetime value.

Part 3: Expansion path

Create a visible reason to grow: usage-based tiers (price steps that rise as the customer uses more, such as per 1,000 records processed) or add-on modules (optional paid extras) tied to the milestones above, plus in-product prompts when a team approaches a limit. Check that the segment can use the add-on; if the buyer never touches the product, a user-level prompt will not work.

Measuring success at 90 and 180 days

Customers cannot be observed past month six within 90 days if they are new, so start with those who are in months 3 to 5 now and assign a random holdout (a randomly chosen group that does not get the new treatment and serves as the comparison).

Contraction is revenue lost from customers who stay but pay less, and expansion revenue is extra revenue from existing customers (more seats, upgrades, add-ons). Report ARPU separately: right-sizing can lower ARPU and still raise lifetime value, so judge on revenue retained over the following months, not on ARPU alone.

Trade-offs and pitfalls

• Targeting only customers about to churn selects the ones hardest to save; include a broader cohort.
• Discounting to retain can cap expansion; prefer value fixes first.
• A small segment means noisy 180-day results; report intervals (a range such as 55% to 75%, instead of a single number, that shows how uncertain a small sample is) and avoid declaring victory on a handful of customers.

Direct answer

A strong answer is one concrete story told in order: the situation and the metric, who I worked with and why, the specific change, how we measured it with a control, the result, and one honest lesson. The model below uses a scheduling product and an activation metric. The numbers are illustrative, constructed so the arithmetic can be checked; use your own real figures when you answer.

Situation

New customers of a scheduling app were signing up but few were getting to value. Activation, the share of new accounts that reach the first moment of real value within a set window, was defined here as booking a first real appointment within 7 days of signup, and it stood at 31% of new accounts. Support tickets told a consistent story: people did not know how to connect their calendar, and the customer-success (CS) team, who onboard larger accounts by hand, saw the same pattern in calls. I was the product manager (PM) for onboarding.

Task and who I worked with

My goal was to raise 7-day activation. I brought three groups in early, each with something I did not have:

• CS knew which setup steps people got stuck on and had call notes from about 20 recent accounts.
• Support pulled the top ticket reasons by count.
• Data analysis (a partner on the data team) built the funnel, with setup steps as stages, and was the one who would verify the effect.

Action: the change

The funnel showed the largest drop between "calendar connection started" and "calendar connection finished". CS confirmed that the permission screen from the calendar provider confused people. We changed three things: a short in-app explainer shown before the provider screen, a default working-hours template so the user was not facing an empty schedule, and a CS-written email, sent on day 1 to anyone who had not finished connecting, with a booking link for a 15-minute help call (for accounts above a size threshold only, to keep CS load manageable).

How we knew it worked

We did not compare before and after, because signups vary by week. We randomised new accounts 50/50 between the new flow and the old one. Out of 1,500 accounts per arm, 465 activated in the control arm (31%) and 540 in the new flow (36%), a difference of 5 percentage points (540 minus 465 is 75 accounts, and 75 / 1,500 is 5%). The 5-point lift was statistically significant. A two-proportion test pools the arms (1,005 of 3,000, or 33.5%), gives a standard error of 1.72 points and a z-score of 0.05 / 0.0172 = 2.9, which is a p-value of about 0.004 (the chance of a gap this large if the new flow did nothing). The 95% confidence interval, the range of lifts consistent with the data, runs from about 1.6 to 8.4 points, so the lift is real but its size is uncertain. We also checked guardrails (other measures that must not get worse while the main one improves): support tickets per 100 accounts were 12.0 in control and 11.6 in the new flow, so they did not rise, and 30-day retention for the activated users was 78% in control and 79% in the new arm, in line, which showed that we were not activating people who then left.

Result

After the result, we shipped the flow to everyone and CS adopted the day-1 email as a standing playbook step. The 5-point lift is the measured number; I would say plainly that the contribution of each of the three changes was not separated, since they shipped together.

What I would do differently

Ship the three changes as separate arms (or in sequence) so we could tell which one did the work, and involve CS in defining the activation metric rather than inheriting it. I would also schedule a 90-day check on retention, since our initial read stopped at 30 days.

Pitfalls in telling this story

• Giving only the team's activity ("we collaborated closely") without the specific decision you made.
• A before/after claim with no control, or a result with no stated denominator.
• Claiming credit for the whole result of a team effort. Say what you did: framed the problem, set the metric, ran the prioritisation, and brokered the CS and support input.