Growth, Activation and Retention Questions

Direct answer

Run a randomised controlled experiment: at signup, assign each new user (or each workspace, for team products) at random to the old or the new onboarding, with one pre-declared primary metric, a few guardrail metrics, a sample size computed in advance, and a fixed run length that covers whole weeks and the full time the primary metric needs to mature. Sample ratio mismatch checks and sequential methods, covered below, refine that core of metric, randomisation, sizing and whole-week runs. For this redesign the primary metric would be the activation rate: the share of new users who reach a defined first-value action within 7 days.

1. What to measure

• Primary metric (decide before launch): activation within 7 days, where "activated" is a behaviour that predicts retention (for example "created a first project and invited a teammate"). One primary metric prevents picking whichever number looks best afterwards.
• What else to watch:
  • Guardrails (things that must not get worse): onboarding completion time, support tickets per new user, crash rate, and signup-to-first-session drop-off.
  • Diagnostic metrics: step-by-step funnel conversion, to see where the effect arises.
  • Longer-horizon outcomes: 30-day adoption of the key feature (the share of users using it at least X times in 30 days), and 60 or 90-day retention. Lifetime value (total gross profit a customer brings) is too slow to be the decision metric, so use activation as the early proxy and check later that activated users in the new arm retain at least as well. If activation rises but retention does not, the redesign pushed people through a step without delivering value.
• Instrumentation first: log the assigned arm, the exposure time, every onboarding step event and the activation event, with user and workspace IDs, before launch. Unlogged churn and adoption events cannot be recovered afterwards.

2. Assignment

Randomise at the unit that experiences the change. For consumer products that is the user: hash the user ID with an experiment-specific salt (run the ID plus a fixed per-experiment string through a hash function, which gives a number that looks random but is always the same for that user) into 50/50 buckets (for example, even result goes to the old flow, odd to the new) at first exposure, so a user always sees the same version. For team products where one person's onboarding affects teammates, randomise by workspace (cluster) to avoid contamination (users in one arm influencing users in the other, such as a teammate sharing the new flow), and treat the workspace as the unit in the analysis. Check for sample ratio mismatch (the arms receiving a different share of users than designed, tested with a chi-square test, which asks how likely a split this uneven is by chance alone); a mismatch means something broke in assignment and the results should not be trusted.

When comparing a product-led guided flow with a customer-success-led (human-assisted) onboarding, add that as a third arm only if capacity allows, and compare activation and cost per activated account, since human onboarding costs real money per account.

3. How large

For a proportion metric the per-arm size is:

$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \,[p_1(1-p_1) + p_2(1-p_2)]}{(p_2 - p_1)^2}$

where $p_1$ is the control rate, $p_2$ the rate you want to be able to detect, $\alpha$ the false-positive rate (0.05) and $1-\beta$ the power (0.80, the chance of detecting a real effect of that size). For these two choices the normal-curve values are z for 1 - alpha/2 = 1.96 and z for 1 - beta = 0.84. Worked by hand for 40% to 43%: (1.96 + 0.84)^2 = 7.85; the spread term is 0.40 x 0.60 + 0.43 x 0.57 = 0.4851; the squared gap is 0.03^2 = 0.0009; so n = 7.85 x 0.4851 / 0.0009, about 4,231 per arm.

import math
from statistics import NormalDist

def n_per_arm(p_control, p_treat, alpha=0.05, power=0.80):
    z = NormalDist().inv_cdf
    z_alpha = z(1 - alpha / 2)   # two-sided test
    z_beta = z(power)
    variance = p_control * (1 - p_control) + p_treat * (1 - p_treat)
    return math.ceil((z_alpha + z_beta) ** 2 * variance / (p_treat - p_control) ** 2)

print("40% -> 43%:", n_per_arm(0.40, 0.43), "users per arm")
print("40% -> 44%:", n_per_arm(0.40, 0.44), "users per arm")
total = 2 * n_per_arm(0.40, 0.43)
print("total users:", total, "days at 600 eligible signups/day:", round(total / 600, 1))

Printed output:

40% -> 43%: 4231 users per arm
40% -> 44%: 2387 users per arm
total users: 8462 days at 600 eligible signups/day: 14.1

Assumptions: baseline activation 40%, the smallest improvement worth shipping is 3 percentage points (the minimum detectable effect), and 600 eligible signups per day (illustrative). The result is most sensitive to the minimum detectable effect: asking for 4 points instead of 3 cuts the requirement from 4,231 to 2,387 per arm, because required size scales roughly with 1 over the effect squared. Choose the effect from the business value of a point of activation, not from what is cheap to detect.

4. How long

• Enrolment needs 8,462 users, which is 14.1 days of traffic. Round up to whole weeks (weekday and weekend users behave differently), so enrol for 3 weeks (12,600 users in this example).
• Then wait for the primary metric to mature: a 7-day activation window means the last enrolled user needs 7 more days, so analysis happens at day 28.
• Do not peek and stop early on a good-looking p-value (the probability of seeing a gap this large if the redesign actually did nothing); that inflates false positives. Decide the end date in advance, or use a sequential method (one whose significance threshold is adjusted so repeated looks stay valid).
• A novelty bump (users react to anything new) fades, so check that the effect in week 3 resembles week 1. Retention and the 30-day adoption outcomes are read later, on the same users, as secondary results.

5. Decision rule

Ship if the primary metric improves with confidence above the agreed bar (for example, the 95% confidence interval for the difference, the plausible range of the true gap, lies entirely above zero) and no guardrail has materially worsened; do not ship if guardrails fail even when the primary improves; if it is flat, the interval tells whether a meaningful effect is ruled out. Segment cuts (new versus returning, device) are exploratory unless pre-registered (written down before launch).

Pitfalls

Picking the metric after seeing results; underpowered tests that "show no effect"; assigning by session instead of by user; contamination between teammates in a workspace; measuring activation but never checking retention.

Direct answer

I would first decide who is worth chasing, then segment by why they left, send different messages through the channels they already accept, make a narrow and conditional offer rather than a blanket discount, and measure the programme against a randomised holdout: a random slice of lapsed customers who receive nothing, so you can see how many return anyway. It "paid for itself" if the incremental contribution (profit after variable costs) from customers who came back only because of the programme exceeds its cost.

1. Who is worth chasing (re-qualification)

Drop or deprioritise before any message goes out:

• Customers who left for reasons we will not reverse: payment fraud, abuse, a chargeback (a payment dispute raised with the bank), or a company that closed.
• Anyone who unsubscribed or has no consent for the channel.
• Customers who never reached value (never activated). They may be better served by an onboarding offer than a win-back.
• Accounts whose cost to serve (support, infrastructure and payment fees) exceeded their contribution.

Then rank the remainder by prior value (tenure, usage depth, plan) and recency of lapse.

2. Segments by reason for leaving

Example messages, three segments:

• Price: "Last spring you paused your plan. We now have a plan at $9/month for the features you used most. Your saved projects are still here."
• Dormant: "Your workspace has been quiet since March. Here is a five-minute setup for the reports most teams start with, and your data is untouched."
• Missing feature: "You asked us for calendar sync. It shipped on 12 August. Reconnect and see your old schedules inside it."

Channels: email first (consent-based), in-app message the moment they log in, push only if still opted in, and paid retargeting audiences (lists of past customers uploaded to an ad platform so they see your ads) if the platform and privacy rules allow, with a suppression list (people to exclude, such as those who have already returned) so they do not see ads. Cap contact at three touches over three weeks.

3. Offers: what I would and would not do

• Would: a one-time, time-limited offer for the price segment only; a free re-activation that restores saved data; a downgrade or pause instead of a goodbye.
• Would not: a blanket discount to everyone, an offer to the missing-feature segment (it does not address their reason, and it gives away margin), or a deep discount without an expiry. Offering a discount teaches customers that cancelling earns one (the discount-precedent risk: next quarter's active customers learn to cancel and wait). Keep the offer rule public to the team, one per customer per 12 months.

4. Did it pay for itself

The discount goes only to the price segment, so I would measure that segment directly (the other segments get no discount and cost only send time; they are read the same way against their own holdouts). Hold out a random 10% of the eligible price-segment list who get nothing. Illustrative numbers: 10,000 eligible, 9,000 treated and 1,000 held out. Reactivation in 30 days: 4.0% treated, 1.5% held out. Contribution per reactivated customer: $12 monthly price x 80% gross margin (the share of revenue left after the direct cost of serving the customer; take the real figure from finance) x 5 months retained (an assumption here; in practice take it from how long previously reactivated customers stayed) = $48. The offer is one month at 50% off ($6 given up per reactivator). Fixed programme cost (design, send, team time): $1,500.

treated, rate_t, rate_h = 9000, 0.04, 0.015
reactivated = treated * rate_t            # 360
would_return = treated * rate_h           # 135 returned anyway
incremental = reactivated - would_return  # 225
value_each = 12 * 0.80 * 5                # 48 dollars contribution
discount_cost = reactivated * 6           # discount goes to ALL 360 who return
cost = discount_cost + 1500
print(reactivated, would_return, incremental)
print(round(incremental * value_each), round(cost), round(incremental * value_each - cost))

This prints 360.0 135.0 225.0 and 10800 3660 7140: 225 incremental customers worth $10,800, against $3,660 of cost, for a net of +$7,140. Note the discount cost covers all 360, including the 135 who would have returned anyway: that is the real price of an untargeted offer, and why I restrict it. The result depends most on retention months per reactivator (5 here): at 2 months the value falls to $19.2 per customer and incremental contribution to $4,320, which is only $660 above the $3,660 cost. So track 90-day survival of the reactivated, not just the reactivation.

Pitfalls

• Counting every return as programme credit. The holdout shows 135 would have returned anyway.
• Treating a day-30 return as success when many reactivated customers churn again in a month.
• Messaging someone who left over a billing dispute with a "we miss you" email.

Direct answer

The viral coefficient (K) is the number of new users each existing user brings in: K = invites sent per user x share of invites that become users. A spike in invites with weak signups is a quality problem, not a growth win. I would stop counting raw signups as the goal and pay out only on activated referrals, add layered fraud signals (device, velocity and social-graph patterns, meaning who invited whom), simplify the flow for honest sharers, and monitor a quality-weighted K (activated referred users per inviter) that falls when the programme is gamed.

Why raw K misleads (illustrative example)

The activated counts come from activation rates of 30% of 300 signups (90) before and 10% of 450 signups (45) after. Raw K rose 50% (0.30 to 0.45) while quality-weighted K halved. Quality-weighted K keeps the idea of K, new users per existing user, but counts only referred users who reach the first key action, so it falls when signups are fake or low value. A dashboard showing only invites and raw K would congratulate the team during an abuse spike.

Cross-functional plan

Fraud detection heuristics

1. Velocity: invites per user per hour and per day well above the normal range; many signups from one device, IP range or payment instrument.
2. Signup quality: disposable email domains, signups that never complete the first key action, many accounts created and abandoned within minutes.
3. Reward-chasing patterns: referred accounts that go quiet right after the reward threshold is passed.
4. Social-graph signals: draw users as dots and invitations as arrows. A legitimate referral graph is mostly trees: each user has one inviter, with chains (A invites B, who invites C) and stars (one user invites many) but no loops. Abuse leaves different shapes: reciprocity (A invites B and B invites A back), a high clustering coefficient (the share of a user's contacts who are also connected to each other, normally near zero in a referral tree), and unusual connected-component size (a connected component is a group of accounts where each can be reached from any other by following invitations in either direction; here, one with more links than a tree would have). The code below computes two simpler stand-ins for the same tight-cluster shape, reciprocal edges and edges per node within each component, rather than the clustering coefficient itself:

from collections import defaultdict

# (inviter, invitee) referral edges: a legitimate tree plus a ring and a mutual pair
legit = [("u1","u2"),("u1","u3"),("u2","u4"),("u2","u5"),("u3","u6"),("u7","u8")]
ring  = [("f1","f2"),("f2","f3"),("f3","f4"),("f4","f1"),("f1","f3"),("f3","f1")]
edges = legit + ring
E = set(edges)

reciprocal = sum(1 for (a, b) in E if (b, a) in E)
print("reciprocal directed edges:", reciprocal, "of", len(E))

adj = defaultdict(set)
for a, b in E:
    adj[a].add(b); adj[b].add(a)
seen, comps = set(), []
for n in adj:
    if n in seen: continue
    stack, comp = [n], set()
    while stack:
        x = stack.pop()
        if x in comp: continue
        comp.add(x); stack.extend(adj[x] - comp)
    seen |= comp
    edges_in = sum(1 for (a, b) in E if a in comp)
    comps.append((len(comp), edges_in))
for size, e in sorted(comps, reverse=True):
    print(f"component size {size}, edges {e}, edges per node {e/size:.2f}")

Output: reciprocal directed edges 2 of 12; component size 6 with 5 edges (0.83 edges per node), size 4 with 6 edges (1.50), size 2 with 1 edge (0.50). How the code works: E holds each invitation once. The reciprocal count checks, for every (a, b), whether (b, a) also exists. The adj dictionary lists each account's neighbours in both directions, and the stack loop starts from an unseen account and keeps adding neighbours until the whole component is found. edges_in counts the invitations whose inviter lies in that component, and the last loop prints size, edge count and edges per node. A tree of n nodes has n-1 edges, so the ring (4 nodes, 6 edges, with a reciprocal pair) stands out. These signals prioritise review; none should auto-ban alone, because families and flatmates also form tight clusters. Velocity and signup-quality checks are the cheap ones to ship first; graph signals come later and exist to prioritise manual review.

Flow changes that help legitimate invites

• Share from a moment of success with a personalised, pre-filled message and a link that shows who invited the receiver; avoid "invite all contacts" defaults that create blast spam.
• Let senders choose a few close contacts and show which of their contacts are not yet users.
• Give the receiver a clear, honest landing page with the value they get, so conversion rises from real interest.

Incentives that favour quality

Pay the reward only when the friend completes the first key action, add a hold period (a delay before credit can be spent, so fraud can be caught first), give a larger reward when the friend is still active at day 30, cap monthly rewards per sender, and prefer in-product value (a premium feature) over cash so the incentive attracts users, not arbitrageurs (people who farm rewards for profit). If fraud rises, throttle the reward before you remove the programme.

Metrics to monitor referral health

Quality-weighted K, invite-to-signup conversion, share of referred signups who activate, day-30 retention of referred vs other users, reward cost per activated referral, flagged-pair rate, share of rewards reversed, and percentage of signups from the top 1% of senders (concentration). Set alert thresholds from the programme's own history rather than from an industry rule of thumb.

Trade-offs

Tighter rules protect budget and also delay rewards for honest users, so measure complaint volume and appeal outcomes. Roll changes out as experiments with a holdout so a drop in referrals can be attributed to the change.

Direct answer

NRR (net revenue retention) is the revenue today from the customers you had 12 months ago (or at the start of the period), including upsells (existing customers buying more or a higher tier) and other expansion, minus contraction (customers who stay but pay less, for example by dropping seats) and cancellations, divided by what they paid at the start. Mid-market here means customers in the middle of the size range (for example, roughly 50 to 500 employees, or a middle band of annual contract value), with the exact cut taken from the business's own segment definition. A +10 point goal in six months is aggressive, so I would frame it as a portfolio of experiments across two levers, churn prevention and in-product expansion, with a target split, named owners, and a dated measurement plan. The brief does not mention human-led account management, so I assume it continues at its current level and design every new lever as product, pricing or lifecycle messaging; if customer success capacity can be added, it becomes a third lever and the split below would change.

Define it and decompose the 10 points

$NRR = \frac{MRR_{start} + Expansion - Contraction - Churn}{MRR_{start}}$

MRR (monthly recurring revenue) is the monthly subscription revenue. Illustrative baseline on 1,000,000 USD starting MRR: expansion 110,000, contraction 30,000, churn 80,000.

NRR = (1,000,000 + 110,000 - 30,000 - 80,000) / 1,000,000 = 100%. GRR (gross revenue retention, which ignores expansion) = (1,000,000 - 30,000 - 80,000) / 1,000,000 = 89%.

Target: 110%. One decomposition: expansion 110,000 to 160,000 (+5 points), contraction 30,000 to 20,000 (+1 point), churn 80,000 to 40,000 (+4 points). New NRR = (1,000,000 + 160,000 - 20,000 - 40,000) / 1,000,000 = 110%; GRR becomes 94%. Total: 5 + 1 + 4 = 10 points. I would hold this as a hypothesis: first measure where baseline revenue loss is concentrated and rebalance. Mapping the roadmap below to the 10 points (illustrative): usage-limit prompts +2 and the new add-on +3 give the 5 points of expansion; the cancellation flow +1 and the health score +2 plus seat-activation +1 give the 4 points of churn; seat-activation also removes 1 point of contraction. Timing caveat: the add-on (weeks 8 to 20) and seat-activation (weeks 12 to 24) land late in the window, so by month six only part of their effect shows in revenue; the 10 points are best read as the run-rate once all five initiatives are live, which is why the trailing-versus-run-rate question under the pitfalls must be settled with leadership first.

Prioritised roadmap

Experiment framing

• Target segment: mid-market accounts, which have enough volume per cohort for tests; if too few accounts for a clean test (a cohort is a group of accounts that started in the same period), test at the user level and read at the account level.
• Hypothesis example: "Showing the upgrade prompt at the limit raises monthly expansion MRR by X% versus control."
• Measurement: randomised holdout at account level (a random share of accounts that do not get the change, kept for comparison); primary metric incremental NRR contribution (expansion minus churn difference); guardrails (metrics that must not get worse): support contacts, downgrade rate, and satisfaction.
• Cadence: monthly readout. NRR on a 12-month basis moves slowly, so track the leading components monthly (expansion, contraction, churn as a share of starting MRR) and compute a trailing NRR for the report.

Trade-offs and pitfalls

• Six months is short for a number whose full definition looks twelve months back. Agree upfront whether leadership means the trailing figure (NRR measured over the last 12 months, the usual definition) or the run-rate (the latest pace annualised), otherwise the goal may be impossible by construction. Illustration: if NRR is 100% today and from now on every month runs at a pace that annualises to 110% (monthly factor 1.1^(1/12) = 1.0080), after six months the trailing 12-month figure blends six old months and six new: 1.0080^6 = 1.049, about 104.9%. The run-rate would read 110% while the trailing figure cannot exceed about 105%, so a six-month goal of 110% trailing would be impossible even with a perfect run.
• Expansion raises NRR but not GRR; if all the gain comes from a few large accounts, it is fragile. Report concentration.
• Price rises can lift NRR and raise churn later; do not count them as a product win without watching churn.
• Dependencies (billing, metering, packaging approval) are the usual cause of slippage; list them first.

Direct answer

Churn is the rate at which customers or their revenue leave over a period. I would distinguish at least four splits: voluntary versus involuntary, customer count (logo churn: each customer account counts as one "logo", whatever it pays) versus revenue, gross versus net revenue, and early versus late in the customer's life. They matter because each has a different cause and a different owner. The easiest to fix is involuntary churn, meaning customers lost to failed payments rather than a decision to leave.

The kinds of churn

A denominator rule keeps these honest: churn for a period is customers (or revenue) lost during the period from those who were active at the start, divided by the number (or revenue) at the start. New customers acquired inside the period do not belong in the denominator.

Worked example (illustrative)

Start of month: 1,000 subscribers at 20 USD per month, so 20,000 USD monthly recurring revenue (MRR: the subscription revenue the business expects to receive every month).

• 30 cancel by choice: voluntary logo churn 30 / 1,000 = 3.0%.
• 12 lose service after payment failures: involuntary logo churn 12 / 1,000 = 1.2%.
• Total logo churn: 42 / 1,000 = 4.2%.
• Revenue lost to those 42 cancellations: 42 x 20 = 840 USD (4.2% of MRR).
• 20 customers downgrade from 20 to 10 USD: contraction of 20 x 10 = 200 USD (1.0%).
• Gross revenue churn: 840 + 200 = 1,040 USD, or 5.2% of 20,000 USD.
• Existing customers upgrade, adding 300 USD of expansion.
• Net revenue churn: (1,040 - 300) / 20,000 = 3.7%.

Same month, three headline "churn" numbers (4.2% logo, 5.2% gross revenue, 3.7% net revenue) plus a 3.0% and 1.2% voluntary and involuntary split. Quoting one without its definition is how teams argue past each other.

Why the distinction changes the action

Voluntary churn is a decision: you need to learn the reason (exit surveys, usage before cancellation) and respond with product or pricing. Involuntary churn is an accident: the customer did not decide anything, so recovering the payment recovers the customer. Revenue versus logo tells you whether to worry about many small customers or a few large ones. Gross versus net stops expansion from masking a leaking base. Early versus late tells you whether the problem is acquisition and onboarding or long-run value.

Which are easiest to fix

1. Involuntary churn: mechanical, measurable and low-risk (retry schedules, automatic card updating, emails before expiry, a grace period). It is a fix to the process rather than a change to the customer's mind.
2. Early voluntary churn from poor onboarding: next easiest, because the cause is usually visible in the first-week funnel and under your control.
3. Late voluntary churn (a competitor, a changed need, budget cuts) is hardest, because much of it lies outside the product.

Pitfalls

Counting a paused or downgraded customer as active without saying so; changing the definition of "active" for free users mid-year; reporting only net revenue churn; including new customers in the denominator.