Conversion Funnel Optimization Questions

Direct answer

Moving free-to-paid conversion from 2% to 5% is a 150% relative increase, too large to bet on any single test, so the right deliverable is a sequenced, 6-month roadmap of experiments, front-loaded with the highest-confidence, lowest-effort changes to build early evidence and momentum, escalating toward bigger, riskier swings once the cheap wins are exhausted. Every experiment in the sequence needs its own stated hypothesis, primary metric, and success threshold, and the roadmap as a whole needs an honest, explicit check on whether the cumulative target is actually achievable in the stated window, which is a distinct and necessary part of the answer, not just the list of experiments itself.

Structured elaboration

Why sequencing, not just a list, matters. Running experiments in priority order rather than in parallel or arbitrary order does two things a flat list does not: it lets each experiment's real result inform whether the next one's underlying assumption still holds (an onboarding-friction fix that under-performs suggests the team's theory of what's blocking conversion needs revision before investing in a bigger, related bet), and it produces early, credible wins that justify continued investment in the later, more expensive experiments to skeptical stakeholders.

Sequencing logic used below. Early months prioritize changes that are cheap to build, fast to measure, and target a well-understood friction point (signup and first-session experience) where a positive result is likely and the downside of a negative result is small. Later months escalate to changes that require more engineering investment, take longer to reach a valid read (a trial-length experiment cannot conclude faster than the trial itself), or touch acquisition mix rather than only in-product behavior. This is the same confidence-versus-effort logic behind formal prioritization frameworks like RICE (reach, impact, confidence, effort) or ICE (impact, confidence, ease); here that logic is applied directly to produce an ordering, not re-derived as its own separate framework.

Alternative ways to arrive at a similar sequence, worth naming briefly. The same prioritized-experiment-sequence approach applies directly if the target metric were a B2B activation rate moving from 20% to 35% instead of a freemium-to-paid rate moving from 2% to 5%: the sequencing LOGIC (cheap and high-confidence first, expensive and uncertain later) does not change, only the specific experiments shift toward B2B-relevant activation levers (a guided admin setup wizard, a dedicated onboarding call for larger accounts) instead of freemium in-product prompts. A second, equally valid way to derive a sequence is customer-journey mapping: lay out every touchpoint from signup through billing (signup, first login, first key action, the upgrade moment, checkout) and design one experiment per touchpoint, rather than ranking by confidence and effort; this tends to produce a similar set of experiments organized by WHERE in the journey they sit rather than by WHEN to run them, which is a useful alternate lens when communicating the roadmap to stakeholders who think in journey stages.

Worked example

The month-6 sample-size calculation, worked explicitly (compact version: applied directly below using the standard two-proportion formula, without re-deriving it from first principles). Using the standard two-proportion sample-size formula

with $p_1 = 0.08$, $p_2 = 0.11$, $z_{\alpha/2} = 1.9600$ (5% two-sided significance), and $z_{\beta} = 0.8416$ (80% power), the computed sample size is 1,499 users per arm (2,998 total), actually computed in Python rather than estimated:

import math
from statistics import NormalDist
nd = NormalDist()
p1, p2, alpha, power = 0.08, 0.11, 0.05, 0.8
z_alpha, z_beta = nd.inv_cdf(1 - alpha/2), nd.inv_cdf(power)
p_bar = (p1 + p2) / 2
n = ((z_alpha*math.sqrt(2*p_bar*(1-p_bar)) + z_beta*math.sqrt(p1*(1-p1)+p2*(1-p2)))**2) / (p2-p1)**2
print(f"n per arm (raw) = {n:.2f}, ceil = {math.ceil(n)}, total = {2*math.ceil(n)}")

Output (actually executed): n per arm (raw) = 1498.39, ceil = 1499, total = 2998. This is what actually determines whether month 6 is even a realistically completable experiment: if this segment has materially fewer than roughly 3,000 eligible new users across the month, the experiment as designed cannot reach a conclusive read in the available time, and either the effect size being tested for needs to be larger (accepting the risk of missing a real but smaller effect) or the experiment needs to run longer than the single month slotted for it in this roadmap.

Trade-offs and pitfalls

• The arithmetic reality of the target deserves explicit acknowledgment. Moving from 2% to 5% free-to-paid conversion is a 150% relative increase; even optimistic individual experiments in a reasonably mature product typically produce single-digit-to-low-teens percent RELATIVE lifts each, not 150% in aggregate from six experiments. A senior answer states this tension directly rather than presenting the roadmap as guaranteed to hit the target: either the current 2% baseline reflects specific, large, fixable frictions (plausible if the product is young or has known major UX gaps, in which case bigger individual lifts are genuinely achievable), or the 5% target itself needs to be revisited as the roadmap's early results come in, with the sequencing designed to surface that reality quickly (front-loading experiments) rather than only in month 6.
• Each experiment's success threshold should be tied to what the CUMULATIVE roadmap needs to add up to the target, not chosen independently per experiment; a set of individually "successful" experiments that each clear their own small threshold can still fall well short of the 2%-to-5% goal if nobody checked the arithmetic adds up across all six.
• A common mistake is running all six experiments as fully independent, non-interacting bets; several of these (the upgrade prompt, the personalized emails, the paywall redesign) touch overlapping parts of the same conversion moment and can interact (a redesigned paywall changes the context the upgrade prompt is shown in), so later experiments in the sequence should account for the compounding effect of earlier shipped changes rather than being designed and analyzed in isolation as if the product were unchanged.
• The app-store-listing experiment (month 5) is measurement-different from the others in a way worth flagging: because it changes acquisition mix rather than in-product behavior, its "success" metric is necessarily cohort-based (comparing users acquired before versus after the change) rather than a randomized A/B split of existing users, which is a weaker causal design and more vulnerable to confounds like seasonality; treat its result with correspondingly more caution than the randomized in-product experiments.

Direct answer

Design the checkout-abandonment test around a single primary metric (checkout completion rate) with named guardrails (refund rate, payment failure rate, revenue per visitor), a sample size sized off the primary metric's baseline rate and minimum detectable effect, a measurement window long enough to cover a full weekly cycle plus any post-purchase guardrail lag, deterministic-hash bucketing so users are consistently assigned and isolated from other concurrent experiments, and a staged rollout (5% then 50% then 100%) gated on both the primary metric and every guardrail staying healthy at each stage.

Structured elaboration

Getting to the hypothesis before designing the test. A test does not start from "let's A/B test the checkout button"; it starts from a diagnosed problem: pull the funnel drop-off data for the checkout step, segment it (device, payment method, new versus returning), and find where drop-off concentrates. If mobile users abandon at 2x the desktop rate specifically at the payment-form step, the hypothesis becomes concrete and falsifiable, not "make checkout better" but a specific, testable causal claim, shown next.

Hypothesis. "Reducing the mobile checkout payment form from 3 required fields to 1 (with autofill) increases mobile checkout completion rate, because the current form's field count is the primary abandonment driver for mobile users at that step."

Primary and guardrail metrics. Primary: checkout completion rate (users who reach checkout_start and reach purchase, the metric the hypothesis is directly about). Guardrails, metrics that must NOT get worse even if the primary improves: refund/chargeback rate within 7 days of purchase, payment failure/decline rate, average order value, and page load time on the checkout step. Guardrails exist because a change can win on the primary metric while quietly damaging something else the primary metric cannot see; the concrete case worth naming explicitly: a simplified, lower-friction checkout can raise completion rate while ALSO raising refund and fraud rates, because some of the friction being removed (a required billing-address confirmation, for example) was doing real fraud-prevention or informed-consent work, not just slowing people down. A test that only looks at the primary metric would ship a change that looks like a clear win and is actually trading real revenue for a fraud or return-rate problem that shows up weeks later. Refund rate needs its own, LONGER measurement window than the primary metric for exactly this reason (see Measurement window below).

Sample size, approach (not full derivation). Sample size for a proportions test like this depends on four inputs: the baseline conversion rate (checkout completion today), the minimum detectable effect (the smallest lift worth being able to detect, decided by business judgment, not statistics, typically informed by the revenue value of that lift versus the cost of running the test longer), the desired significance level (commonly $\alpha = 0.05$), and the desired statistical power (commonly $1-\beta = 0.80$). Those four feed a standard two-proportion power calculation to produce the required sample size per arm; this answer names the inputs and the approach at a conceptual level, since the point here is knowing which four factors determine sample size and how they trade off against each other, not re-deriving the closed-form formula step by step. Two practical notes beyond the inputs: (1) the baseline rate should come from RECENT data for the exact segment being tested (mobile checkout specifically, not the blended overall rate), since a mismatched baseline silently under- or over-powers the test; (2) if traffic is limited, a smaller minimum detectable effect requires disproportionately more sample, so the MDE choice is often the practical lever that determines whether the test is even feasible in a reasonable timeframe.

Measurement window. Primary metric: at least one full week (to average out day-of-week effects like weekday-versus-weekend purchase behavior), and typically extended to cover 2 full weekly cycles for a stable read. Guardrails on a LAG, like refund rate, need a window that extends PAST the point where the last user entered the test, since a refund can happen up to the refund policy's window (7, 14, or 30 days) after purchase; declaring the test a winner from primary-metric data alone, before the guardrail's own lag window has closed, is how the conversion-versus-refunds tension above gets missed in practice.

Segmentation rules. Pre-register the segments you will check (mobile versus desktop, new versus returning, geography) BEFORE launching, not after seeing results; this is a bias-prevention rule, since hunting for a significant segment after the fact (the primary metric was flat, but "mobile users in Region X" moved) inflates the false-positive rate through unplanned multiple comparisons. If mobile is the hypothesized driver, mobile can be the PRIMARY analysis population rather than just a post-hoc segment, with desktop as a secondary check that the change did not regress the segment it was not designed for.

Rollout plan. Staged: 5% of eligible traffic for an initial health check (catch a broken build or an obviously bad guardrail signal fast, cheaply), then 50% once nothing looks wrong, then 100% once the test reaches its planned sample size and duration and the primary metric plus every guardrail clears their bar. Each stage gate is a genuine decision point, not a formality: a guardrail regression at 5% should halt the rollout before it reaches more users, not just get noted for later.

Worked example: novelty effects and personalization-decay

A test of a PERSONALIZED checkout flow (recommended payment method pre-selected based on past behavior) surfaces a pitfall generic to any personalization change, worth calling out explicitly rather than folding into the generic guardrail list above: a novelty effect can make week-1 results look better than the change's true steady-state impact, because some of the lift is users reacting to something new and different rather than to the personalization genuinely being more relevant. The concrete failure mode: a personalization model trained on stale behavioral data can also DECAY in relevance over time as user preferences drift, so a test that looks flat-to-positive in week 1 and week 2 can quietly erode by week 6 as the underlying model's recommendations grow less accurate relative to current behavior, an effect a short measurement window would never catch. The mitigation is to extend monitoring PAST the initial test window into a post-launch tracking period specifically watching for a downward trend in the personalized arm's lift over time, not just a single end-of-test readout, and to re-train or re-validate the personalization model on a defined cadence so a stale model is caught before its degraded relevance shows up as a silent conversion-rate regression.

Trade-offs and pitfalls: running many concurrent experiments (the experiment-platform question)

This question's own ask, how to handle users in multiple concurrent experiments and prevent cross-contamination, is a real system-design problem once a company runs more than a handful of tests at once, and deserves concrete technical treatment, not just a mention:

• Deterministic hash-based bucketing. Assign each user to a variant via a hash of (user_id, experiment_id) (for example hash(user_id + experiment_id) mod 100), NOT a random draw at request time. This gives two properties that matter: the SAME user always lands in the SAME variant across sessions and devices as long as user_id is stable (consistency), and, critically, hashing user_id combined with the experiment's OWN id means a user's bucket in experiment A is statistically INDEPENDENT of their bucket in experiment B, because the two hashes are effectively unrelated draws even for the same user. This independence is what allows many experiments to run simultaneously on overlapping populations without the assignments correlating with each other.
• Overlapping-audience management via layers/namespaces. A common pattern (used by large-scale experimentation platforms) organizes experiments into orthogonal LAYERS: within one layer, experiments are mutually exclusive (a user in the checkout-form-layout experiment cannot simultaneously be in a different checkout-form-layout experiment), while experiments in DIFFERENT layers (checkout layout versus a pricing-page experiment) can run concurrently on the same user, since they touch unrelated parts of the product and their effects are assumed roughly additive/independent. This lets a platform run dozens of tests at once without every pair needing manual conflict analysis.
• Explicit interaction checks for experiments that plausibly interact. Orthogonal hashing prevents ACCIDENTAL correlation, but it does not prevent two experiments from genuinely interacting in effect (a checkout-form simplification test and a pricing-display test both touching the same screen could interact even with independent bucketing). For experiments the team suspects might interact, either force them into the same layer (mutual exclusion) or run an explicit 2x2 factorial analysis on the overlap population to check whether the combined effect differs from the sum of the individual effects.
• Cross-contamination beyond assignment: leakage through shared infrastructure. Bucketing correctly is necessary but not sufficient; a shared cache, a shared recommendation model retrained on mixed-variant data, or a support agent manually "fixing" a treatment-group user's cart back to the control experience are all ways cross-contamination can happen AFTER correct assignment. Auditing for this means checking that variant-specific behavior does not leak into shared state that both arms read from.

Other pitfalls: running the test on a stage-X restatement (payment-entry drop-off, or onboarding/activation instead of checkout) uses this exact same framework unchanged, only the hypothesis, primary metric, and guardrails swap to match the new stage, nothing about the sample-size approach, bucketing, or rollout staging logic changes. Common mistake: treating the 5% rollout stage as a formality rather than a real gate, shipping straight to 50% because the initial numbers "look fine" after only a few hours, well before the guardrails (especially the lagged refund-rate guardrail) have had time to produce a meaningful read.

Direct answer

First-touch credits the very first channel a user ever interacted with, last-touch credits the channel immediately before conversion, linear splits credit evenly across every touchpoint, and time-decay weights touchpoints closer to conversion more heavily. For a subscription product with a long time-to-purchase and multiple touchpoints, time-decay is the best default: it reflects that a touchpoint six weeks before signup plausibly still mattered (unlike last-touch, which ignores it entirely) while not pretending it mattered exactly as much as the touchpoint from yesterday (unlike linear).

Structured elaboration

The four models, what each one actually claims:

Why time-decay for THIS scenario. A subscription product with a long time-to-purchase (weeks to months) and multiple touchpoints is exactly the case where first-touch and last-touch both fail badly, in opposite directions: first-touch would credit, say, a single early blog-post visit for a conversion that only actually happened after three retargeting ads and a comparison-page visit weeks later; last-touch would credit only the final retargeting ad and completely erase the blog post's role in creating awareness in the first place. Linear treats a single passive ad impression as equally causal as an active demo request, which overstates the passive touches. Time-decay's exponential weighting is a reasonable middle ground: it still credits the early blog post (unlike last-touch), but proportionally less than the touches that happened closer to the actual signup decision (unlike linear), which better matches how influence plausibly decays over a multi-week consideration window.

Effect on budget allocation. Attribution model choice is not a reporting footnote, it directly changes where marketing spend flows, because teams optimize toward whatever the attribution model says is working: first-touch attribution systematically pushes budget toward awareness/top-of-funnel channels (content, SEO, display) at the expense of channels that actually close deals; last-touch does the opposite, starving upper-funnel investment because it never gets credit, which can quietly shrink the pipeline that bottom-funnel channels depend on, since retargeting has nothing to retarget once the top of the funnel dries up. Time-decay's more balanced credit distribution tends to produce more balanced budget allocation across the funnel, though it still requires an explicit, defensible decay half-life (how many days until a touch's credit halves), and that parameter is itself a judgment call that shapes the result.

Worked example

Practical deduplication before attribution can even run. Before any of the four models can assign credit, the touchpoint sequence itself has to be built correctly, and the real-world wrinkle is that the SAME user shows up as multiple session-level identifiers across channels and devices: an anonymous cookie ID on a laptop for the first blog visit, a different cookie ID on a phone for a retargeting-ad click, and finally a logged-in user_id at signup. Attributing credit per session-level ID instead of per resolved USER would double-count touchpoints (the same real journey looks like two separate people, each getting first-touch credit) and would completely miss the cross-device journey that time-decay is supposed to be crediting. The practical fix is an identity-resolution step BEFORE attribution: stitch session IDs to a stable user identity via deterministic keys where available (login, email hash) and probabilistic signals where not (same IP plus similar timing plus device fingerprint, used cautiously and only to supplement deterministic matches), then run the attribution model over the resolved, deduplicated touchpoint sequence, not the raw session log.

A concrete decayed-credit example. A user's resolved journey before signing up for a subscription: blog post visit 30 days before signup, comparison-page visit (from a paid search ad) 10 days before signup, retargeting ad click 2 days before signup, then signup. Using an exponential time-decay with a 7-day half-life (credit halves every 7 days further from conversion):

where $t$ is days before conversion. Raw weights: blog ($t{=}30$) $= 2^{-30/7} \approx 0.0513$; comparison page ($t{=}10$) $= 2^{-10/7} \approx 0.3715$; retargeting ad ($t{=}2$) $= 2^{-2/7} \approx 0.8203$. Normalizing so the three weights sum to 100% of credit: total $\approx 1.2431$, so blog gets $0.0513/1.2431 \approx 4.1\%$, comparison page gets $0.3715/1.2431 \approx 29.9\%$, and retargeting gets $0.8203/1.2431 \approx 66.0\%$. Contrast with last-touch, which would give the retargeting ad 100% and the blog post and comparison page 0%, completely erasing two of the three touchpoints that built toward the decision.

Trade-offs and pitfalls

• Attribution-window LENGTH is a related but distinct parameter choice from the model itself, and needs its own decision. Even a well-chosen model (time-decay) is applied over some bounded lookback window (how far back a touchpoint is even eligible to be considered), commonly 1, 7, or 30 days for e-commerce and often 30 to 90 days for subscription products with longer consideration cycles. A short window (1 day) systematically undercounts a long-consideration-cycle product like this one, since most of the journey above (the blog post 30 days out) would fall entirely outside a 1-day window and get zero credit regardless of which model is used. A long window (say 180 days) risks over-crediting stale, no-longer-relevant touches, a display ad seen six months ago that the user has long forgotten, inflating that channel's apparent influence. The window should be set based on the product's OBSERVED typical time-to-purchase distribution, not a default copied from a different product category.
• Common mistake: picking a single attribution model as if it were ground truth rather than a modeling choice with a stated bias, and then making irreversible budget cuts based on it. A more defensible practice is to run multiple models side by side (or a proper causal method like incrementality testing via geo-holdouts) and treat convergent signal across models as higher confidence than any single model's number.
• Multi-touch models still assume the touchpoints you tracked are the whole story. Offline influence (word of mouth, a colleague's recommendation, a conference talk) never appears in the touchpoint log at all and gets zero credit under any of these models, systematically undercounting brand and community channels that are genuinely hard to instrument.
• Time-decay's half-life is a hidden lever. A very short half-life converges toward last-touch behavior; a very long half-life converges toward linear. Reporting a time-decay number without disclosing the half-life used is nearly as opaque as not naming the model at all, since the parameter choice can swing the credit distribution as much as the model choice itself.

Direct answer

No single attribution model is right for every use case, so the actual deliverable is a decision: heuristic models (linear, time-decay) are cheap, transparent, and good enough for a quarterly report a non-technical stakeholder needs to trust at a glance; regression-based and Shapley-value models are data-driven and defensible under scrutiny but expensive to build and maintain, appropriate when real budget decisions ride on channel comparisons; and Markov chain modeling sits in between, data-driven and computationally tractable enough to refresh frequently, which makes it the strongest candidate for near-real-time optimization despite giving up some of Shapley's theoretical guarantees.

Structured elaboration

Recommendation. For a quarterly marketing report, where the audience needs to trust the number quickly and the cost of a slightly less statistically rigorous credit split is low, a heuristic model (time-decay, since it at least reflects recency better than pure linear credit) is the right default: it is fast to produce, easy to explain and defend in the room, and does not require maintaining a fitted model across reporting periods. For real-time bidding or budget-reallocation optimization, where the credit split directly drives a repeated, automated decision and the cost of a systematically biased heuristic compounds every cycle, the Markov chain approach is the strongest fit: it is genuinely data-driven (unlike the heuristic), fast enough to refresh on the cadence an optimization loop needs (unlike Shapley at meaningful channel-set sizes), and its removal-effect credit responds to real shifts in user behavior automatically as the transition matrix is re-estimated. Regression-based and Shapley-value models are best reserved for a periodic, deeper validation exercise, run quarterly or less often, to check whether the operational Markov-based credit split still roughly agrees with a more rigorous method, rather than as the always-on production system themselves.

Markov chain attribution, in depth

The model. Represent every distinct marketing channel as a state, plus a Start state and two absorbing states, Conversion and Null (drop-off without converting). Estimate the transition matrix $P$ directly from observed touchpoint sequences: $P_{ij}$ is the empirical fraction of times a user at state $i$ moved next to state $j$.

Absorption probability via the fundamental matrix. Partition $P$ into the transient states (Start plus every channel) and the two absorbing states, giving the standard block form

where $Q$ is the transient-to-transient sub-matrix and $R$ is the transient-to-absorbing sub-matrix. The fundamental matrix

gives, entry by entry, the expected number of visits to each transient state before absorption, and

gives, for every transient state, the probability of eventually being absorbed at Conversion versus Null. The overall baseline conversion probability is $B$'s entry for the Start state and the Conversion column.

Removal effect: turning absorption probability into channel credit. To credit channel $c$, rebuild the transition matrix with every transition that would have gone INTO $c$ redirected instead to Null (the touch simply does not happen, so that user falls out of the funnel at that point), recompute the Start-state conversion probability under this modified chain, and take the difference from the baseline. Formally, if $p^*$ is the baseline conversion probability and $p^*_{-c}$ is the conversion probability with channel $c$ removed, channel $c$'s removal effect is $p^* - p^*_{-c}$. Normalizing every channel's removal effect to sum to 1 and multiplying by the total observed conversions gives each channel's credited conversion count, which is the standard way this method is applied in practice.

Sparse-channel handling. A channel with very few observed touches produces an unstable transition-probability estimate (a handful of paths through it can push its outgoing probabilities to extreme, noisy values), and removing it can produce an unrealistically large or small removal effect purely from sampling noise. Two practical fixes: apply additive (Laplace-style) smoothing to the raw transition counts before normalizing into probabilities, which pulls a sparse channel's estimated transitions toward the overall average rather than trusting a handful of observations completely, or group genuinely low-volume channels into a single "Other" state before estimating the chain at all, sacrificing per-channel granularity for a materially more stable estimate on the channels that matter.

Algorithmic complexity. Building the transition-count matrix is a single linear pass over the touchpoint data, $O(\text{total touches})$. The fundamental-matrix computation is a $k \times k$ matrix inversion, $O(k^3)$ with standard dense linear algebra, where $k$ is the number of transient states (channels plus Start); computing the removal effect for every channel means repeating this inversion once per channel, so the full removal-effect attribution run is $O(k^4)$ in the worst case. This stays entirely practical for a realistic channel count (tens of channels, not thousands), which is exactly why the Markov approach can be refreshed far more often than a Shapley computation whose cost grows exponentially in the same $k$.

Worked example

Ten pinned synthetic user paths across three channels (Search, Social, Email), actually computed with numpy linear algebra rather than hand-derived:

import numpy as np

paths = [
    (["Search", "Social"], "Conversion"), (["Search"], "Conversion"),
    (["Social", "Email"], "Conversion"), (["Email"], "Conversion"),
    (["Search", "Social"], "Null"), (["Social"], "Null"),
    (["Search", "Email"], "Conversion"), (["Email", "Social"], "Conversion"),
    (["Search"], "Null"), (["Email"], "Null"),
]
STATES = ["Start", "Search", "Social", "Email", "Conversion", "Null"]
idx = {s: i for i, s in enumerate(STATES)}
n = len(STATES)

counts = np.zeros((n, n))
for seq, outcome in paths:
    prev = "Start"
    for ch in seq:
        counts[idx[prev], idx[ch]] += 1
        prev = ch
    counts[idx[prev], idx[outcome]] += 1

P = counts.copy()
row_sums = P.sum(axis=1, keepdims=True)
for i in range(n):
    P[i, i] = 1.0 if row_sums[i] == 0 else P[i, i]
    if row_sums[i] > 0:
        P[i] = P[i] / row_sums[i]

def conversion_probability(P):
    transient = [i for i, s in enumerate(STATES) if s not in ("Conversion", "Null")]
    absorbing = [idx["Conversion"], idx["Null"]]
    Q = P[np.ix_(transient, transient)]
    R = P[np.ix_(transient, absorbing)]
    N = np.linalg.inv(np.eye(len(transient)) - Q)
    B = N @ R
    return B[transient.index(idx["Start"]), 0]

baseline = conversion_probability(P)
print(f"Baseline P(eventually Convert | Start) = {baseline:.4f}")

def remove_channel(P, channel):
    P2 = P.copy()
    c = idx[channel]
    for i in range(n):
        if i != c and P2[i, c] > 0:
            P2[i, idx["Null"]] += P2[i, c]
            P2[i, c] = 0.0
    return P2

removal_effects = {}
for channel in ["Search", "Social", "Email"]:
    p_removed = conversion_probability(remove_channel(P, channel))
    effect = baseline - p_removed
    removal_effects[channel] = effect
    print(f"Remove {channel}: P(Convert) drops to {p_removed:.4f}, removal effect = {effect:.4f}")

total_conversions = sum(1 for _, outcome in paths if outcome == "Conversion")
sum_effects = sum(removal_effects.values())
print()
print("Normalized channel credit (removal effect / sum of removal effects):")
for channel in ["Search", "Social", "Email"]:
    norm = removal_effects[channel] / sum_effects
    credited = norm * total_conversions
    print(f"{channel}: normalized credit = {norm:.4f}  ->  credited conversions = {credited:.2f}")

Output (actually executed):

Baseline P(eventually Convert | Start) = 0.6000
Remove Search: P(Convert) drops to 0.3208, removal effect = 0.2792
Remove Social: P(Convert) drops to 0.3400, removal effect = 0.2600
Remove Email: P(Convert) drops to 0.2600, removal effect = 0.3400

Normalized channel credit (removal effect / sum of removal effects):
Search: normalized credit = 0.3175  ->  credited conversions = 1.91
Social: normalized credit = 0.2957  ->  credited conversions = 1.77
Email: normalized credit = 0.3867  ->  credited conversions = 2.32

The baseline conversion probability of 0.6 exactly matches the raw empirical rate (6 of the 10 pinned paths converted), which is the expected sanity check for a first-order Markov chain estimated directly from this data. Email receives the highest removal effect and the highest credited-conversion share (2.32 of the 6 total conversions), consistent with Email appearing in three of the six converting paths including as the sole touch in one of them, while still discounting for the fact that Search and Social also carry real, non-zero contribution.

Trade-offs and pitfalls

• The Markov chain's removal effect is a modeled counterfactual estimated from observational data, not a causal experiment; a channel that happens to be touched mostly by users who were already likely to convert for unrelated reasons will still show a large removal effect, since the model has no way to separate correlation from causation on its own.
• Shapley's exponential cost is a real, not theoretical, blocker at scale: attributing credit across dozens of individual campaigns rather than a handful of channel groups makes exact Shapley infeasible, and Monte Carlo approximation trades exactness for tractability in a way that needs its own convergence checking, which is easy to skip under time pressure.
• A common wrong turn is picking one model and reporting its output as ground truth rather than as one estimate among several reasonable ones; running two different models (a heuristic and the Markov chain, for example) side by side and treating a large disagreement between them as a signal worth investigating, rather than picking whichever number is more convenient, is closer to how a senior analyst actually uses these outputs.
• The first-order Markov assumption (the next state depends only on the current state, not the full history) is a real approximation; a channel that behaves very differently depending on what came before it (a retargeting ad that only works after an initial organic visit, for instance) will not be captured correctly without moving to a higher-order chain, which increases both data requirements and the state-space size sharply.

Direct answer

Drop-off is a user failing to advance to the NEXT step of a bounded, ordered sequence they already entered; churn is an already-engaged, already-converted user ceasing to use the product over an OPEN-ended, recurring relationship. Both concepts can be defined three distinct ways, time-based, activity-based, and cohort-based, and the six resulting definitions below are genuinely different measurements, not restatements of each other: time-based definitions use a fixed clock window, activity-based definitions use behavior itself (a session boundary or a specific action) rather than a fixed window, and cohort-based definitions aggregate to a group-level rate rather than flagging individuals.

Structured elaboration

The six required definitions, drop-off first, then churn, each with SQL-like pseudocode:

Definition 1, drop-off, time-based (SQL-like pseudocode):

SELECT user_id
FROM step_n_events sn
WHERE NOT EXISTS (
    SELECT 1 FROM step_n1_events sn1
    WHERE sn1.user_id = sn.user_id
      AND sn1.event_ts BETWEEN sn.event_ts AND sn.event_ts + INTERVAL '7 days'
);

Definition 2, drop-off, activity/session-based (SQL-like pseudocode):

-- session_id assigned upstream via a 30-minute inactivity-gap rule
SELECT sn.user_id, sn.session_id
FROM step_n_events sn
WHERE NOT EXISTS (
    SELECT 1 FROM step_n1_events sn1
    WHERE sn1.user_id = sn.user_id AND sn1.session_id = sn.session_id
);

Definition 3, drop-off, cohort-based (SQL-like pseudocode):

SELECT cohort_week,
       COUNT(DISTINCT n.user_id) AS reached_step_n,
       COUNT(DISTINCT n1.user_id) AS reached_step_n1,
       1.0 - COUNT(DISTINCT n1.user_id) / COUNT(DISTINCT n.user_id) AS dropoff_rate
FROM step_n_events n
LEFT JOIN step_n1_events n1 ON n1.user_id = n.user_id
GROUP BY cohort_week;

Definition 4, churn, time-based (SQL-like pseudocode):

SELECT user_id, MAX(event_ts) AS last_active_ts
FROM all_usage_events
GROUP BY user_id
HAVING MAX(event_ts) < CURRENT_DATE - INTERVAL '30 days';

Definition 5, churn, activity/behavioral-based (SQL-like pseudocode):

SELECT user_id
FROM key_action_events
GROUP BY user_id
HAVING MAX(event_ts) < CURRENT_DATE - INTERVAL '14 days'
   -- the window for the CORE action is often shorter than the general
   -- inactivity window in definition 4, since a user can still be logging in
   -- (satisfying definition 4's activity bar) while having quietly stopped
   -- doing the thing that actually signals engagement
;

Definition 6, churn, cohort-based (SQL-like pseudocode):

SELECT signup_cohort_month,
       COUNT(DISTINCT CASE WHEN subscription_status = 'canceled' THEN user_id END)
         AS churned_users,
       COUNT(DISTINCT user_id) AS cohort_size,
       1.0 * COUNT(DISTINCT CASE WHEN subscription_status = 'canceled' THEN user_id END)
         / COUNT(DISTINCT user_id) AS cohort_churn_rate
FROM subscriptions
GROUP BY signup_cohort_month;

The conceptual differences that make these genuinely two different phenomena, not the same idea at two scopes. Drop-off's time windows are short (hours to a few days), because it measures failure to complete a BOUNDED sequence that is supposed to happen close together in time. Churn's time windows are long (weeks to months), because it measures disengagement from an OPEN-ended, ongoing relationship with no natural next step to fail at. Drop-off has no EXPLICIT "I am dropping off" event, it is always inferred from the absence of the next step; churn, for contractual/subscription products, sometimes DOES have an explicit event (a cancellation), which is a fundamentally more reliable signal than any inactivity-based inference, since it removes the guesswork of picking a threshold.

Worked example

A concrete scenario where picking the wrong one of the three definition types for CHURN specifically produces a materially different number: a B2C app with 10,000 users in a signup cohort. Using time-based churn (no usage event in 30 days): 2,200 users churned (22%). Using activity-based churn (no CORE action, defined as creating content, in 14 days, even though many of these users still opened the app to browse): 3,600 users churned (36%), a meaningfully higher rate, because it catches users who are technically still "active" by a login-based definition but have functionally disengaged from the product's actual value. Using cohort-based subscription churn (only counting users who explicitly canceled a paid plan): 850 users churned (8.5%), the lowest of the three, because it only counts the subset of the 10,000 who were even ON a paid plan to begin with and explicitly canceled, silently excluding free users who quietly stopped using the product without ever having a plan to cancel. All three numbers are "correct" by their own definition; reporting any one of them as "the churn rate" without naming which definition produced it is how two teams end up in a meeting disagreeing about a number that was never actually the same metric.

Trade-offs and pitfalls

• Which definition fits which business scenario. Time-based drop-off is the right tool for near-real-time funnel-step debugging and A/B test monitoring, since it produces a fast, individual-level signal. Activity/session-based drop-off is the right tool for diagnosing single-session UX friction, where the steps are supposed to happen back-to-back. Cohort-based drop-off is the right tool for trend reporting and cross-segment comparison on a dashboard, not for real-time alerting on any one user. Time-based churn is the right tool for triggering simple, automatable re-engagement campaigns (a "we miss you" email at day 30). Activity/behavioral churn is the right tool for early-warning and predictive churn modeling, catching disengagement before the harder time-based threshold fires, especially valuable in B2B contexts where a customer-success team wants to intervene before a renewal decision, not after. Cohort-based churn (particularly the explicit, contractual version) is the right tool for board-level revenue reporting (MRR, monthly recurring revenue, churn and logo churn), since it ties directly to realized revenue impact rather than an inferred behavioral proxy.
• Common mistake: picking a single churn threshold (30 days) and applying it uniformly across products with very different natural usage cadences. A daily-habit product (a messaging app) genuinely churning at 30 days of inactivity is a different signal than a quarterly-use product (a tax-filing app) where 30 days of inactivity is completely normal and says nothing about disengagement; the threshold should be calibrated against the product's own typical usage rhythm, not borrowed from a different product category.
• Drop-off definitions inferred purely from absence carry irreducible ambiguity that an explicit churn-cancellation event does not. A user who "dropped off" at checkout might complete the purchase tomorrow, a week from now, or never; without a hard cutoff, EVERY drop-off number is provisional until enough time has passed, exactly the right-censoring problem familiar from survival analysis: an observation whose true outcome has not yet occurred by the measurement cutoff is neither a confirmed success nor a confirmed failure, only not yet resolved.