Attribution & Conversion Measurement Questions

Approach: treat the campaign as a measurable funnel from source → trial sign-up → paid conversion → revenue, instrument end-to-end, compute unit economics, and validate with cohort analysis and sensitivity tests for finance.

Instrumentation

• Capture UTM params (utm_source, utm_medium, utm_campaign, utm_content) on landing pages and persist to a tracking cookie/session.
• On sign-up, tie UTMs to user_id in DB and analytics (Mixpanel/GA4/Amplitude + backend events).
• Record first_touch, last_touch, and multi-touch event timestamps; record payment events and revenue (gross, discounts, MRR/ARR).
• Push cost data per campaign (ad spend, creative, agency fees) into analytics via ETL.

Key metrics to compute

• CAC = total campaign spend / new customers attributed (use chosen attribution model).
• Trial conversion rate = paid_conversions / trial_signups.
• Time-to-convert = median/mean days from trial_start → first_payment.
• LTV = projected NPV of expected gross margin per customer (use cohort retention & ARPU).
• Payback period = CAC / monthly gross margin.

Attribution windows & model

• Primary: 30-day trial window for trial → paid; 90-day revenue window for early churn.
• Use first-touch for acquisition credit and data-driven multi-touch (weighted) for revenue attribution; present both.

Cohort analysis

• Build cohorts by campaign, acquisition week, and source. Track conversion, retention (D1, D7, D30, D90), ARPU, churn.
• Compute LTV per cohort using observed retention curves and conservative/optimistic projections.

Presentation to Finance

• Deliver a one-pager: assumptions, funnel KPIs, CAC/LTV, payback, sensitivity table (best/base/worst), and recommended action.
• Include data table + visuals: funnel conversion, cohort retention curves, LTV vs CAC scatter.
• Call out limitations (attribution assumptions, data latency, cohort size) and proposed validation experiments (UTM hygiene, randomized holdout).

Approach summary: combine randomized holdout experiments for ground-truth incremental measurement with a probabilistic cross-device attribution model used for operational reporting and QA. Prioritize privacy-preserving instrumentation that improves signal quality while avoiding deterministic IDs.

1. Clarify success metrics & constraints

• Primary metric: incremental conversions attributable to app ad clicks (lift), measured as conversion lift per campaign or cohort.
• Constraints: no deterministic device graph, GDPR/CCPA, limited PII, low latency not required for offline reporting.

2. Measurement strategy

• Ground truth via randomized holdouts: For a subset of ad impressions/clicks, randomly assign users (at campaign or geo-day cohort level) to “exposed” vs “holdout” groups at the app-click moment. Ensure holdouts are large enough for power; run tests per campaign or aggregated by publisher to estimate Average Treatment Effect on the Treated (ATT). Use intent-to-treat and instrumented assignment to avoid contamination (e.g., server-side buckets).
• Probabilistic cross-device attribution: Build a ML model that links app-click cohorts to web conversions probabilistically using aggregated hashed signals (IP-range bucket, device OS+version, coarse geo (city/zip3), time-of-day, browser fingerprints hashed, app version). Model predicts probability that a web conversion belongs to an app-click cohort using Bayesian or logistic regression with calibrated output; use it to attribute fractional conversions for reporting.

3. Instrumentation changes

• Server-side click logging: persist click events with campaign_id, publisher_id, timestamp, anonymized fingerprint hash (one-way salted HMAC per time-window), coarse geo, app_version, publisher_click_id if available (hashed), and experiment_bucket.
• Web conversion tagging: collect same hashed fingerprint features at conversion time and record conversion value, timestamp, and experiment_bucket if present (e.g., via same salted hash derived from browser signals).
• Use rotating salts and time-windowed hashing to reduce re-identification risk while enabling matching.
• Add event-level metadata to support join windows (e.g., click -> conversion attribution window) and mark first-touch vs last-touch.

4. Privacy & compliance

• No persistent cross-site identifiers; one-way salted hashes rotated per X days and per product environment.
• Aggregate outputs at cohort level with minimum cell sizes; apply differential-privacy noise when exposing small cohorts.
• Store raw linking features only in secure, access-controlled environments for model training; purge per retention policy.

5. Validation & calibration

• Use randomized holdout ATT as ground truth to evaluate and calibrate probabilistic model — compare predicted fractional attribution vs measured lift; adjust model bias with calibration factors.
• Run sensitivity analyses: different hashing windows, feature subsets, attribution windows.
• Report uncertainty: publish confidence intervals and bias estimates for each campaign-level attribution.

6. Operationalization & roadmap

• Short-term (0–3 months): add server-side click logging and web conversion hashed features; pilot randomized holdouts on high-volume campaigns.
• Medium (3–6 months): train probabilistic model, integrate fractional attribution into BI with calibration weights from holdouts, add privacy-preserving aggregation.
• Long-term (6–12 months): expand holdouts across publishers, automate continuous calibration, explore privacy-enhancing tech (federated learning or secure multi-party computation) for publisher-supplied signals.

Trade-offs and risks

• Holdouts give unbiased lift but cost revenue and require buy-in from marketing.
• Probabilistic matching improves coverage but can misattribute — mitigated by calibration to holdouts and exposing uncertainty.
• Stronger privacy (shorter hash windows, more aggregation) reduces matching accuracy; balance by testing sensitivity and increasing holdout coverage to anchor estimates.

Key deliverables for PM

• Experiment plan with power calculations and holdout allocation.
• Instrumentation spec for engineering and privacy review.
• Dashboard showing calibrated attributed conversions, lift from holdouts, and confidence intervals.
• Roadmap milestones and stakeholder communication plan to manage short-term revenue impact from holdouts.

Overview: Move from last-touch to weighted multi-touch attribution (MTA) for bookings across web, mobile app, and email using an event-level, deterministic-first then probabilistic model. Phased rollout: instrumentation & schema alignment → attribution engine development → parallel reporting → stakeholder validation → full cutover.

Metric definitions:

• Booking: unique booking_id with status (confirmed) and timestamp.
• Touch: user_id (hashed), device_id, channel, campaign_id, event_type (view/click/open), timestamp.
• Attributed bookings: booking-level revenue distributed across touches per attribution model (linear, time-decay, position-based).
• Primary metrics: attributed bookings, attributed revenue, touch-through rate, conversion lag.

Required event joins:

• Join booking events to preceding touches by user_id/device_id within lookback window (e.g., 30 days) using deterministic keys; where missing, fallback to probabilistic joins by IP+user-agent+time fuzzy match.
• Enrich with campaign metadata and channel taxonomy.

Deduplication strategies:

• Normalize identifiers, hash PII, dedupe duplicate events by event_id and dedupe touches within session windows (e.g., 30 min).
• For cross-device duplicates, use deterministic cross-device mapping where available (login_id); otherwise accept model-level noise and surface confidence scores.

Storage & compute:

• Store raw events in append-only data lake (partitioned by date/channel), curated event tables in warehouse.
• Attribution compute: batch Spark/Beam jobs for windowed joins and attribution scoring (daily), incremental streaming for near-real-time needs.
• Estimate: join-heavy jobs; plan for cluster autoscaling, cost controls, and materialized aggregates (by campaign/date) to avoid repeated heavy recompute.

Validation steps:

• Shadow run MTA alongside existing last-touch for 4–8 weeks; compare totals, channel shares, and anomalies.
• Run reconciliation tests (bookings count, revenue sum), sample-level audits (trace individual booking attribution), A/B tests on reporting decisions, and sensitivity analysis on lookback and weight parameters.

Stakeholder communication:

• Create a one-pager explaining model, business implications, and expected shifts in channel attribution; include examples.
• Present phased timeline, KPIs to monitor, and rollback criteria.
• Provide dashboards showing both last-touch and MTA concurrently, plus confidence bands and action guidance for marketing teams.
• Train stakeholders on interpretation and update budgeting/commission rules with a cooling period (e.g., 3 months) before operational decisions.

This plan balances technical fidelity, cost, and business trust via parallel reporting, strong validation, and clear stakeholder education.

Framework overview:

1. Clarify objectives & constraints

• Goal: maximize net LTV/CAC over 12–24 months while preserving growth velocity and testability.
• Constraints: minimum spend commitments, measurement latency, channel operational limits.

2. Data & modeling strategy (measure + infer)

• Instrumentation: unify event-level data (clicks, impressions, affiliate IDs, organic source, user_id, cohort_time, revenue events) in a warehouse; ensure deterministic identifiers and conversion windows.
• Attribution backbone:
  • Use multi-touch probabilistic approaches: fit a Markov-chain path model to estimate conversion probabilities and remove last-click bias; compute channel removal effects to get channel contribution shares.
  • Complement with Shapley-value attribution on high-value cohorts to validate marginal contribution.
  • Build time-to-conversion survival (Kaplan–Meier) curves per channel/cohort to account for time-lag and discount future revenue (e.g., monthly decay or 5–10% discount).
• Incrementality: run controlled experiments (holdout panels / geo experiments) and targeted spend ramps to measure true lift and validate model-derived contributions.

3. LTV per channel computation

• For each user cohort by first-touch channel and month:
  • Estimate expected discounted gross revenue over horizon using survival-adjusted retention and average order value.
  • Allocate multi-touch contributions across channels using Markov/Shapley adjusted shares.
  • LTV_channel = sum over users (discounted revenue × channel_share) / number_of_acquired_users_attributed_to_channel.
• CAC_channel = total spend driving attributable acquisitions (use experiment-corrected conversion uplift where available) / attributed acquisitions.
• Net LTV/CAC = LTV_channel / CAC_channel (or LTV - CAC for absolute ROI).

4. Phased budget reallocation plan (90-day rolling phases)
   Phase A — Measure & protect (0–6 weeks)

• Freeze structural spend; reallocate up to 10% into measurement (holdouts, geos, analytics) and low-risk tests.
• Run small, statistically powered holdouts and 2–3 controlled spend ramps per channel.

Phase B — Validate & tilt (6–12 weeks)

• Apply model results and experiment signals: shift 10–25% of marginal budget from underperforming channels (low net LTV/CAC and low incrementality) into top channels, keeping safety caps.
• Continue targeted A/B and geo tests for scalability.

Phase C — Scale & monitor (12–24 weeks)

• If experiments confirm predicted LTV uplifts, move additional budget (up to 50% of re-allocable pool) towards high-net-LTV channels with monitoring guardrails (daily CAC drift, weekly cohort LTV checks).
• Maintain 10–15% of budget for ongoing experimentation and to avoid single-channel dependency.

5. Governance & monitoring

• Weekly dashboard: cohort-level LTV, CAC, Net LTV/CAC, incrementality lift, time-to-convert distributions.
• Automated alerting on cohort prognosis deviating >10% vs model.
• Quarterly model refresh and reconciliation with experimental results.

Trade-offs & caveats

• Attribution models imply assumptions; prioritize experiments to de-risk decisions.
• Time-lag means short-term CPA can mislead; emphasize discounted multi-month LTV.
• Maintain diversification to manage vendor/channel risk.

Recommended immediate next steps

• Deploy unified dataset and run initial Markov & survival models on last 12 months.
• Kick off two simultaneous holdout tests (geo and affiliate holdout) within 2 weeks.
• Prepare phased budget playbook and stakeholder sign-off for the 90-day plan.

Goal: attribute DoorDash orders to marketing channels accurately across devices and delayed conversions so marketing can optimize spend while respecting privacy and scale.

High-level approach

• Use a hybrid system: deploy a data-driven (causal) model as the long-term truth for planning and budget allocation, backed by pragmatic rules (time-decay + last-touch) for near-real-time reporting and channel-level dashboards.

Compare approaches

• First-touch: credits the first recorded interaction. Pros: simple, good for awareness metrics. Cons: overweights top-of-funnel, ignores later conversion intents and cross-device friction.
• Last-touch: credits final interaction before conversion. Pros: intuitive for optimization, low latency. Cons: biased toward retargeting, ignores assisted channels.
• Time-decay: splits credit by recency (exponential or log). Pros: balances early and late influences. Cons: arbitrary decay params, still descriptive not causal.
• Data-driven (causal/attribution modeling): uses uplift modeling, multi-touch heterogeneous effects, or Shapley-value–style decomposition with experimentation priors. Pros: estimates marginal impact, better for budget decisions. Cons: needs rich data, compute, and validation via experiments.

Implementation steps

1. Requirements & constraints: define objectives (ROAS, incremental orders), privacy limits (PII minimization, GA4 cookieless), latency needs.
2. Instrumentation: standardize UTM tagging, capture ad logs (impression, click, campaign IDs), server-side event tagging (app open, add-to-cart, order), and deterministic cross-device keys where available (logged-in user IDs).
3. Linkage: build deterministic join on user_id for logged-in sessions; probabilistic/linking fallback using hashed device signals and last-touch heuristics for guest flows while respecting privacy.
4. Interim reporting: implement time-decay attribution with configurable half-life (e.g., 7 days) for operational dashboards.
5. Data-driven model: build uplift models (random forests / causal forest) or use Shapley decomposition on user-level paths aggregated to campaigns. Train on features: exposures, recency, frequency, creative, channel, user propensity, seasonality.
6. Experimentation backbone: design and run holdout and geo/auction-randomized incrementality tests to estimate causal lift per channel/campaign and calibrate model priors.

Required data sources

• UTM parameters, click IDs (gclid, fbclid), campaign metadata
• Ad platform logs (impressions, bids, spend, creative ids)
• Server-side event logs from app/web (user_id, timestamps, events)
• Cookie/device IDs where allowed; hashed identifiers for linking
• Conversion outcomes (orders, lifetime value, cancellations)
• Contextual data: time, location, promotions, store availability

Validation strategy

• Incrementality experiments:
  • Holdout experiments: random user-level holdouts for small campaigns to get unbiased lift.
  • Geo/region experiments for supply-side channels where user-level randomization is hard.
  • Auction-randomized experiments (if supported) to randomize auctions for ad exposure.
• Use experiments to:
  • Measure channel-level incremental ROAS and calibrate model outputs.
  • Validate model assumptions (e.g., decay half-life) and Shapley attributions.
  • Monitor heterogeneity (by cohort, device, new vs returning users).
• Ongoing metrics: predicted vs experimental lift mismatch, stability over time, and business KPIs (orders, AOV, retention). Retrain model when drift exceeds threshold.

Governance & privacy

• Prefer server-side linking on hashed user_id; avoid storing PII in ad logs.
• Implement differential privacy / aggregate reporting for sensitive slices.
• Maintain an experimentation calendar to avoid contamination and coordinate spend.

Outcome

• Short term: time-decay reports for ops; experiments to produce channel-level incrementality estimates.
• Medium term: deploy calibrated data-driven model for budget allocation and bidding, backed by continuous experimental validation to ensure causal accuracy and business-aligned decisions.