Attribution & Conversion Measurement Questions

Approach — high level:

1. Define objective & metrics: incremental revenue (net new $), conversion, ARPU. Segment users (cohort, funnel position).
2. Data sources: randomized experiments (A/B/n), product usage logs (events, timestamps, user IDs), billing records, covariates (user demographics, prior spend).

Experiment-first strategy:

• Run feature-level randomized trials where feasible (encourage/enable feature for treatment). Measure click-to-purchase and revenue lift vs. control to get causal incremental effects.

Incremental modeling for non-randomized features:

• Use uplift/causal inference models (Doubly Robust, Causal Forests) on quasi-experimental exposures (instrumental variables where available). Model predicted counterfactual revenue per user if not exposed.

Handling overlapping features:

• Use factorial experiments (multi-armed) when features interact. In observational modeling, fit additive + interaction terms; use Shapley value–style decomposition (or Hs-Attribution) to fairly allocate overlapping incremental revenue across features based on marginal contributions.

Validation:

• Holdout validation: reserve a time-based or random holdout; compare predicted incremental revenue to observed in fresh experiments.
• Sensitivity analyses: test model to omitted variable bias, alternative specifications, and placebo features (should show zero effect).
• Backtesting: apply model to historical A/B results to recover known lifts.

Uncertainty & confidence intervals:

• For experiments: bootstrap or use t-tests / regression with clustered SEs to get CIs on mean incremental revenue.
• For model-based estimates: use bootstrapped resampling of users or Bayesian posterior intervals from causal models; for Shapley allocations, propagate uncertainty by sampling from the distribution of marginal contributions.
• Report percentiles (e.g., 95% CI), p-values, and minimum detectable effect given sample size.

Reporting & operationalization:

• Dashboard showing point estimates, CIs, sample sizes, assumptions, and interaction flags.
• Include recommended next actions (run factorial test where interaction uncertainty is high).
  This combination yields causal estimates where possible, principled allocations for overlaps, and transparent uncertainty quantification.

Situation: Marketing reports that the recommender, which favors higher‑margin items, may be “cannibalizing” sales of promoted (discounted) items — we need a causal attribution to measure net revenue impact.

Approach (high level)

• Run a randomized experiment where users (or sessions) are randomly assigned to: Control (current recommender) vs Treatment (margin‑aware recommender). Where randomization isn’t feasible, use quasi‑experimental causal methods (instrumental variable, difference‑in‑differences, or propensity‑score–matched synthetic controls) or an uplift/randomized encouragement design if only exposure can be nudged.
• Combine A/B for average treatment effect and uplift modeling for targeted impact (who gains/loses).

Datasets required

• Orders table: order_id, user_id, timestamp, item_ids, quantities, item_prices, discounts, net_revenue
• Catalog: item_id, base_price, cost, margin, promo_flag
• Exposure logs: which recommendation(s) were shown/clicked, rank, page
• User profile & behavior: past orders, segments, tenure, recent sessions
• Marketing/promotion calendar: active promotions by item/time
• Returns/refunds and long‑term LTV data

Modeling choices

• Primary: A/B test with random assignment -> estimate ITT and complier average causal effect (CACE) if not everyone sees/clicks recommendations.
• Secondary: Uplift model (two-model or direct uplift) to identify heterogenous effects for targeting: who increases basket size vs who shifts promo purchases.
• Adjustments: Regression with covariates (pre‑treatment behavior), clustering by user, and pre‑post difference‑in‑differences if rollout is staggered.
• Causal checks: balance tests, SUTVA considerations, pre‑trend tests, and power calculations accounting for skewed revenue distribution (use log or heavy‑tailed robust estimators, or quantile treatment effects).

Attribution logic

• Net revenue impact = change in total net_revenue (including margin), not just AOV, within evaluation window.
• Decompose effect into:
  1. Incremental full‑price / higher‑margin sales
  2. Decline in promo item revenue (cannibalization)
  3. Changes in conversion rate and order frequency
• Compute per‑user and aggregated uplift in margin dollars over a suitable horizon (30–90 days) to capture downstream effects.

Guardrail metrics to report

• Primary: Incremental net margin per user and total incremental margin (with confidence intervals)
• Secondary/guardrails:
  • Conversion rate (orders per active user)
  • Promo item revenue and units sold (absolute and % change)
  • Average order value and basket size
  • Promo cannibalization rate = reduction in promo revenue attributable to treatment
  • Customer churn / repeat purchase rate and 30/90‑day LTV uplift
  • Return/refund rate
  • Exposure→click and click→buy funnel metrics (to detect UI effects)
  • Segment breakouts (new vs returning, high vs low spend)
• Statistical/operational: sample size, p‑values, lift with CIs, and runtime stability (time‑series of daily effect).

Why this approach

• Randomized A/B gives unbiased average causal effect; uplift modeling adds actionability by identifying who to target to maximize net margin while minimizing promo cannibalization. Guardrails ensure we measure both intended benefit and unintended harms (marketing KPIs, retention, long‑term value).

Clarify objective: attribute conversion credit across multiple touchpoints when impressions/events are partially missing and sampling bias exists, so outputs are actionable for channel investment (incremental ROI).

Comparative overview

• Rule-based (last-touch, first-touch, linear):
  • How: deterministic rules assign fixed credit.
  • Assumptions: full observability not required; assumes rule reflects causal importance.
  • Data reqs: minimal (touch sequence and conversion flag).
  • Pros: simple, explainable, fast.
  • Cons: biased when missing touches; ignores interactions and sequence effects → can mislead spend decisions.
• Fractional/time-decay:
  • How: splits credit by fixed weights (recency/decay).
  • Assumptions: pre-specified decay reflects reality; missing data reduces total observed credit.
  • Data reqs: touch timestamps, optionally channel-level priors.
  • Pros: more flexible than rule-based; still transparent.
  • Cons: arbitrary weights, sensitive to missing impressions and sampling bias.
• Data-driven probabilistic (Markov chains, Shapley/coalitional):
  • How: estimate transition/conditional contribution or marginal contribution by channel across paths.
  • Assumptions: observed paths are representative of true behavior (or can be adjusted); Markov assumes limited history dependence; Shapley assumes order-agnostic marginal contributions.
  • Data reqs: many complete user paths (touch sequences to conversion/non-conversion), timestamps, channel IDs; ideally deterministic deduped eventing.
  • Pros: captures interactions, sequence effects, and gives incremental-like metrics (removal effect in Markov; average marginal contribution in Shapley).
  • Cons: sensitive to missing data/sampling; computationally heavier; needs variance estimation.

Handling incomplete data & sampling bias

• Weighting & augmentation: use inverse-probability weighting (IPW) to correct for known sampling probabilities; use propensity models to re-weight sampled users to population.
• Impute missing touches carefully: use channel-level delivery logs, probabilistic imputation where missingness is MCAR/MAR, but avoid over-imputation that invents causal signals.
• Use experiments or holdout tests to calibrate: validate modelled incremental effects against paid incrementality tests (geo or randomized holdouts).
• Uncertainty quantification: bootstrap or Bayesian posterior to get confidence intervals on channel contributions.

Recommended approach for investment decisions

1. Baseline: produce rule-based and fractional outputs for stakeholder transparency.
2. Primary allocation: run probabilistic methods (Markov removal effects and Shapley value) on reweighted/imputed dataset; report expected incremental conversions per channel with CIs.
3. Calibrate: compare model estimates to A/B or geo experiments; if systematic deviation, adjust via scaling factors.
4. Actionable metric: translate incremental conversions to incremental ROAS (marginal revenue – cost) and rank channels by expected ROI and risk (CI width).
5. Operationalize: use model outputs as priors in automated budget optimization, but require periodic re-calibration with fresh experiments and monitor for shifts (new creatives, privacy changes).

Key caveats

• Probabilistic outputs are associative unless validated by experiments — use them for prioritization and hypothesis generation, not as sole proof of causality.
• Always present uncertainty and sensitivity to missingness; recommend investment shifts gradually and verify with holdouts.

Requirements & clarification:

• Output: incremental lift (causal contribution) per channel and per campaign, with confidence intervals, at user- and aggregate-levels, and daily/weekly cadence.
• Constraints: data latency, identity resolution, privacy (PII), minimum exposure sparsity.

High-level pipeline:

1. Ingestion: raw event streams (impressions, clicks, conversions, cost) + CRM (user IDs, conversions offline) + product metadata (campaign, channel, creative) → raw store (S3, partitioned by date).
2. Identity resolution & sessionization: deterministic + probabilistic stitching to user_id → session windows.
3. Exposure timeline builder: for each user, construct time-ordered exposures with timestamps, campaign/channel labels, dose (impr/click), value, and conversion labels with attribution window flags.
4. Feature store: user features (demographics, lifetime value, recency), time-varying covariates (recent exposures counts, time-since-last-exposure), campaign features (bid, creative), temporal features (weekday, seasonality).
5. Modeling infra: train/test splits by time and user; store models + explainability artifacts.
6. Reporting: aggregate lift per channel/campaign with CIs and dashboards.

Data requirements & features:

• Granularity: user × timestamp exposures and conversions.
• Label: conversion within window and revenue; also time-to-event for survival approaches.
• Features: static user covariates, rolling exposure counts per channel/campaign (lags), recency, interaction terms (channel × campaign), price/promotions, seasonality indicators.
• Treatment encoding: multi-dimensional sparse vector per time window (dose per channel/campaign).

Handling time-varying exposures:

• Frame as panel/longitudinal data with time steps (daily). Use lagged treatments and covariates to model carryover and decay.
• Use exposure windows and decays (exponential) as engineered features.
• For modelling, use sequence-aware models: recurrent nets / Transformer encoders for representation, or time-series causal forests.

Model choice:

• Heuristics (last-touch, linear attribution) are baseline but biased.
• Preferred: causal uplift / heterogeneous treatment effect models adapted for multiple, simultaneous treatments:
  • Option A (scalable): Convert to multiple binary treatments and use doubly-robust estimation with inverse probability weighting + multi-task causal forest / X-learner to estimate marginal incremental effect per channel/campaign. Estimate propensity to be exposed (time-varying) using gradient boosting; use DR to reduce bias.
  • Option B (sequence causal): Train a sequence-to-outcome model (Transformer/RNN) that conditions on past exposures and covariates, then use counterfactual rollout: mask exposures for a channel/campaign and predict outcome; use targeted regularization and importance weighting to correct for selection bias.
  • Add Bayesian hierarchical layer to pool across campaigns and obtain CIs.

Validation of incremental estimates:

• Randomized experiments: compare model estimates to A/B test lift where available.
• Backtesting: simulate counterfactual by holdout periods and synthetic interventions (e.g., zeroing a channel) and compare predicted vs observed.
• Placebo tests: check no effect when assigning fake campaign timestamps.
• Overlap & propensity diagnostics: check common support; trim regions with poor overlap.
• Calibration: aggregate modelled incremental conversions to campaign-level and compare to observed experiment lift; compute RMSE, bias.
• Sensitivity analyses: vary unobserved confounding assumptions (Rosenbaum bounds), different windows/decays.
• Uncertainty quantification: bootstrap or Bayesian posteriors; report CIs and minimum detectable effect.

Trade-offs:

• Causal forests + DR: interpretable, fast, good with tabular covariates; needs well-estimated propensity.
• Sequence counterfactuals: capture complex temporal interactions but require heavy compute and careful regularization; more brittle to selection bias.

Operational considerations:

• Retrain cadence (weekly), drift monitoring (treatment propensities, feature drift), compute incremental per-dollar (ROAS lift).
• Privacy: aggregate outputs, differential privacy if required.

This design balances causal rigor, temporal dynamics, and operational scalability to produce per-channel/campaign incremental lift with validated uncertainty.

Requirements & objectives:

• Accurately credit channels (ads, promotions), restaurant partners, referrals across multi-touch journeys with possible delayed conversions.
• Support near-real-time and batch reporting, explainability, experiments validation, and scale to millions of events/day.

Data collection:

• Event-level ingestion: impressions, clicks, promotion views/redemptions, referral codes, order events, restaurant interactions; include timestamps, user_id, device_id, campaign_id, partner_id, attribution_token, client and server-side identifiers.
• Persist touch sequences per user (sessionization, dedupe, TTL for stale touches).
• Ingest offline conversions (call-center, POS) with deterministic joins (order id, phone, referral code) and probabilistic matching (device graphs) plus confidence scores.
• Store feature tables accessible to modeling layer.

Architecture (high-level):

• Streaming pipeline (Kafka) → enrichment (user graph, campaign metadata) → sessionizer → touch-store (timeseries DB / BigQuery / Snowflake) → attribution engine (batch/model serving) → reporting/ML features.
• Compute: distributed Spark/Dataproc for heavy ETL; Flink/Beam for streaming attribution.

Attribution methods & trade-offs:

• Rule-based (last-touch, first-touch, time-decay): simple, fast, explainable; biased for multi-touch and delayed effects.
• Model-based (probabilistic, uplift, survival models): estimate incremental impact, handle time-lag (time-to-conversion models, hazard models), control for confounders, better for long-term ROI. Requires careful feature engineering and causal assumptions.
• Game-theoretic (Shapley): fair, axiomatic credit allocation using marginal contribution; interpretable per-order but expensive: naive Shapley is O(2^n). Use sampling (Monte Carlo Shapley), or weighted/approximate Shapley (Truncated sequences, kernel SHAP-like).

Recommended hybrid approach:

• Use rule-based for realtime low-latency dashboards (e.g., last-click with TTL), and model-based + approximate Shapley offline for billing/strategic decisions.
• Train a conversion model (e.g., gradient boosted trees) to predict probability of conversion given sequence and time; compute marginal contributions by perturbing touches or using SHAP values on the model to approximate Shapley.

Handling offline conversions:

• Deterministic joins when possible; propagate attribution using stored touch sequences and time windows.
• For probabilistic matches, attach match-confidence and propagate uncertainty into credit using weighted contributions.

Computational scaling:

• Precompute sessionized sequences in batch and cache per-user touch windows.
• For Shapley: sample subsets, limit max touches (e.g., last 6), parallel Monte Carlo across cluster, use influence functions or model-agnostic SHAP approximations to reduce calls.
• Use vectorized model scoring and map-reduce for marginal computations; store intermediate results for reuse.

Validation against randomized experiments:

• Run holdout/A/B experiments: randomize exposure to treatments (ads, promos, restaurant placements) at user or region level.
• Compare attribution-derived incremental estimates to experiment-measured lift (ATE); calibrate model outputs to match experimental lift.
• Use backtesting: simulate attribution over past experiment windows; check bias, variance, and confidence intervals.
• Perform sensitivity analyses (vary time windows, matching thresholds, number of Shapley samples).

Metrics & monitoring:

• Track consistency: predicted incremental ROI vs experimental ROI, unexplained conversion fraction, attribution stability over time.
• Monitor latency, pipeline failure rates, match confidence distributions, and cost per attributed conversion.

Governance & productization:

• Version models and attribution rules; document assumptions.
• Expose transparency: per-order attribution breakdown, confidence intervals, and recommended actionability (pause/scale campaigns).
• Align finance/biz stakeholders on which method feeds billing vs optimization.

Key trade-offs summary:

• Latency vs accuracy: real-time rules vs slower model/Shapley pipelines.
• Explainability vs causal validity: rules are clear but biased; model/Shapley more accurate but rely on model/assumption quality.
• Cost vs fidelity: approximate Shapley and sampling provide practical middle ground.

This design balances operational constraints with rigorous causal validation and provides paths to scale and reconcile online and offline conversions.