Growth Metrics & Unit Economics Questions

Attribution approaches for first-purchase revenue

Deterministic rules:

• First-touch: credit channel of first recorded touch. Simple but ignores later influence.
• Last-touch: credit most recent touch before conversion. Easy but overweights late channels.

Fractional approaches:

• Linear: equal credit across touchpoints in user journey.
• Time-decay: more weight to recent touches using exponential decay.

Data-driven multi-touch:

• Uplift modeling: estimate incremental impact of each touch by comparing treated vs counterfactual (requires experiments or quasi-experimental matching).
• Shapley-value or Markov chain attribution: compute marginal contribution across paths (Shapley handles combinatorial fairness; Markov estimates removal effect).

Challenges:

• Data: cross-device identity resolution, cookie loss, lagged events.
• Computation: path explosion for long journeys; needs map-reduce or Spark and path aggregation.
• Interpretation: correlated channels, multicollinearity; advanced models give allocations but require careful business translation.

Recommendation: implement deterministic baselines, run Shapley/Markov for insights, and iterate with uplift experiments on high-value channels.

Situation: We deployed a churn model with strong CV metrics but saw no lift in retention campaigns. Below are five categories of causes, diagnostics, and fixes.

1. Data leakage / label mismatch

• Reason: Training labels use future info (post-churn actions) or different churn definition than campaign targeting.
• Diagnostic: Re-run temporal validation, inspect feature creation windows, compute label distribution across training/production.
• Fix: Enforce strict feature windows; align label definition to the actionable horizon.

2. Population shift / sample bias

• Reason: Training cohort differs from current users (seasonality, new product changes).
• Diagnostic: Compare feature distributions (KS test), calibration drift, PSI.
• Fix: Retrain with recent data, weight samples, or use domain adaptation.

3. Poor causal link / targeting mismatch

• Reason: High predicted churners may be unresponsive to retention offers (e.g., churn due to price vs UX).
• Diagnostic: Analyze uplift via randomized holdout; segment by churn cause if available.
• Fix: Combine churn score with estimated treatment effect; prioritize treatable segments.

4. Measurement and attribution issues

• Reason: Campaign delivery failures, wrong holdout, or short measurement windows.
• Diagnostic: Audit delivery logs, verify randomization, compute intent-to-treat (ITT) and per-protocol effects.
• Fix: Fix tracking, use proper A/B design, extend outcome window.

5. Operationalization and thresholding

• Reason: Thresholds/ binning lead to low precision or low reach; offers insufficient frequency/value.
• Diagnostic: Evaluate precision@k, expected uplift per recipient, cost per treatment.
• Fix: Optimize for business metric (expected lift * cost), run budget-aware targeting.

Validation plan: conduct a randomized experiment with stratified sampling over model score buckets, measure ATE and incremental retention, run subgroup diagnostics, and iterate on feature/label alignment. This approach surfaces whether model or operational failures caused null lift and guides targeted fixes.

Goal: identify users who will increase retention if given a personalized offer (uplift modeling).

Data collection

• Randomized experiment: random assignment to treatment and control across targetable population; log assignment, offer exposure, and outcome (retention). Ensure adequate sample and stratify by key covariates.

Modeling approaches

1. Two-model approach: train separate propensity models for treatment and control (predict retention), compute uplift = P(Y|T)-P(Y|C).
2. Single-model direct uplift: model treatment interaction terms or use meta-learners (S-learner, T-learner, X-learner). X-learner works well with imbalanced treatment sizes.
3. Tree-based uplift models: uplift/random forest (transform features to capture treatment effect heterogeneity).
4. Causal forests / CATE estimators for rigorous HTE estimates; provide uncertainty.

Evaluation metrics

• Qini curve and Qini coefficient: incremental gain vs population ranked by predicted uplift.
• Uplift AUC or AUUC; absolute incremental net retention at top k.
• Use ATE / incremental effect calibration in held-out randomized subset.

Deployment under budget constraints

• Rank users by predicted uplift per cost of treatment; compute expected incremental retention and cost to find optimal cutoff via knapsack-like optimization (maximize incremental retention subject to budget).
• Consider greedy/LP selection using estimated uplift and treatment cost; include uncertainty by penalizing estimates with high variance.

Validation and monitoring

• Run small-scale randomized validation on top-decile to confirm observed uplift.
• Monitor realized incremental retention, calibration drift, and treatment saturation (diminishing returns). Retrain periodically with new experiment data.

Ethics/guardrails

• Ensure fairness; avoid systematically excluding protected groups; track campaign adverse effects. Use conservative thresholds when model uncertainty high.

Situation: At my previous company we launched a freemium analytics feature and saw signups but low paid conversions. Task: I needed to identify whether marketing or product was the blocker and recommend a prioritized change to improve ROI. Action: I defined metrics clearly: CAC = total channel spend / new paying customers; LTV = average revenue per paying user * average lifetime (months); conversion = paid signups / trial signups; retention = % users active at 90 days. I pulled data from analytics and billing systems, validated by cross-checking user IDs between sources, sampling raw events for 200 users, and confirming attribution windows. I segmented CAC by channel and matched LTV by cohort. Findings: Paid-search had CAC = $120, LTV = $300 (LTV:CAC = 2.5), organic had CAC = $20, LTV = $180 (9:1). I recommended shifting 30% of budget from low-efficiency paid-search to retention-focused product work (improve onboarding and feature discoverability) and increasing investment in organic channels. Result: Over the next quarter, blended CAC fell 18%, conversion to paid increased 12%, and projected LTV:CAC rose to ~3.6, improving unit economics. Key learnings: define metrics precisely, validate attribution, and use cohort LTV when making channel decisions.

Last-click vs Multi-touch: pros & cons

Last-click

• Pros: Simple, easy to compute and explain, aligns with last-step conversion credit.
• Cons: Ignores upper-funnel influence, undervalues awareness channels, biases toward conversion-channel tactics.

Multi-touch approaches

1. Linear

• Description: split credit equally across all touchpoints.
• Pros: Simple, credits all contributors; useful when many touches matter.
• Cons: Over-simplifies timing/importance.
• Recommend for: early-stage analysis when no strong evidence about decay; fair baseline for B2B with multiple touchpoints.

2. Time-decay

• Description: weight touches closer to conversion more heavily (exponential or half-life).
• Pros: Reflects recency importance; retains contribution of earlier touches.
• Cons: Requires selection of decay parameter; may still undercount long sales-cycle influence.
• Recommend for: B2B sales-influenced funnels where later touches (demo, trial) are more predictive but earlier marketing seeds pipeline.

3. Data-driven (algorithmic/MD analysis)

• Description: use statistical models (Shapley, uplift models, Markov chains, logistic regression attribution) to allocate credit based on observed impact.
• Pros: Most accurate if data-rich; allocates credit by causal association.
• Cons: Requires large, clean multi-touch datasets; computationally heavier and sensitive to bias.
• Recommend for: mature B2B funnels with CRM+touch logs and sufficient conversions—used to inform strategic budgets and sales enablement.

Recommendation for B2B sales-influenced funnel

• Start with time-decay as pragmatic default (captures recency) while building data infrastructure. As data volume grows, move to data-driven methods (Markov/Shapley) to quantify upstream channel influence on pipeline and closed-won, and report both pipeline-attribution and revenue-attribution separately. Always align attribution windows with sales cycle length and validate with holdout/incrementality tests.