Causal Inference Questions

Direct answer. Measuring cannibalization requires distinguishing orders that would have happened anyway through the merchant's direct channel from orders that are genuinely net-new because of the platform; the identification approach centers on comparing merchants (or markets) with different platform exposure while controlling for their pre-existing direct-channel trajectory, since a naive "platform orders went up, direct orders went down" comparison can't separate substitution from a merchant simply growing overall.

Structured elaboration.

1. Data needed: merchant-level order volume split by channel (platform versus direct) over time, ideally spanning a period before and after the merchant joined the platform (or before and after a platform feature that plausibly shifts cannibalization risk).
2. Cohorting strategy: group merchants by their platform adoption date (a staggered-adoption structure) and compare direct-channel trajectory changes around each merchant's own adoption date, rather than comparing across merchants with very different baseline scale.
3. Metrics: total order volume (platform plus direct combined) as the metric that reveals TRUE net effect; direct-channel volume alone can look like cannibalization even when total volume for the merchant is growing, which would suggest a smaller genuine cannibalization share than a direct-only view implies.
4. Identification approach: a difference-in-differences design comparing each merchant's direct-channel trajectory before and after platform adoption to a group of similar merchants who haven't yet adopted (using the staggered adoption timing as the source of a control group), estimating the causal effect of adoption on direct-channel volume specifically. The share of orders that shifted, rather than being net-new, is then this DiD-estimated direct-channel decline as a fraction of the platform's added order volume for the same merchants.

Worked example. If merchants show, on average, direct-channel orders declining by an estimated 15% relative to their non-adopting-yet peers in the months after platform adoption, while total order volume for those same merchants rises 40% over the same window, the honest interpretation is that a meaningful share of platform growth is net-new demand, with a real but smaller cannibalization component specifically on the direct channel; reporting BOTH numbers (the 15% direct-channel decline and the 40% total growth) is more informative than either alone.

Trade-offs and pitfalls. Staggered-adoption DiD carries the same two-way-fixed-effects bias risk as any staggered rollout when the cannibalization effect itself is heterogeneous across merchant cohorts (a merchant that adopted early during rapid platform growth may show a different cannibalization pattern than one adopting later into a more saturated market), which argues for a modern staggered-DiD estimator rather than a naive two-way fixed-effects regression here.

Direct answer. A lift study (holdout test) withholds a randomly-selected slice of the eligible population from a retention campaign and compares their outcomes to the treated group, giving a clean causal estimate of the campaign's incremental effect. Sizing, timeline, and handling contamination and attrition are the practical design decisions that determine whether the resulting estimate is trustworthy.

Structured elaboration.

1. Treatment assignment: randomize at the user level if campaigns don't have spillover between users, or at a cluster level (household, region) if they might (a referral-driven campaign, for instance, could leak from treated to holdout users if they're connected).
2. Sample sizing: power the holdout to detect the smallest effect size that would still justify the campaign's cost; a holdout that's too small to detect a real but modest effect wastes the whole exercise, since a null result would be ambiguous (no effect, or just underpowered).
3. Timeline: run long enough to capture the campaign's full effect window, including any delayed response, but not so long that the holdout group's natural churn (unrelated to the campaign) swamps the signal; a pre-specified analysis window tied to the campaign's expected mechanism (immediate nudge versus slow-building habit change) avoids post-hoc cherry-picking of when to look.
4. Contamination: monitor whether holdout users are reached anyway through another channel (a separate team's campaign, an unrelated notification that happens to have similar content); a contaminated holdout biases the estimated effect toward zero, understating the campaign's real impact.
5. Attrition: if holdout users are more likely to churn or become unreachable simply from not being treated (differential attrition), track completion rates in both arms and treat a divergence as a data-quality red flag requiring investigation before trusting the final comparison.

Worked example. For a churn-prevention model triggering a retention offer, hold out a random 10% of eligible users from receiving the offer for 8 weeks (long enough to capture both an immediate response and a slower habit-formation effect, based on prior campaigns of similar type). If a QA audit reveals 4% of the holdout group received a similar offer through an unrelated marketing channel, that contamination should be reported alongside the headline lift number, since it means the TRUE effect is somewhat larger than the observed (contaminated) estimate.

Trade-offs and pitfalls. A permanent holdout group is sometimes kept running even after a campaign is proven effective, specifically to detect novelty decay or long-run effect changes that a one-time test would miss; the trade-off is the ongoing cost of deliberately withholding value from that group, which needs an explicit sunset plan or a business justification beyond the initial measurement.

Direct answer. A regression-based plan to move from a raw correlation between marketing spend and sales toward a defensible causal estimate: specify a model that includes spend alongside the confounders you can measure (seasonality, price promotions, competitor actions), test for remaining omitted-variable bias, and validate with out-of-sample or placebo checks before trusting the coefficient on spend.

Structured elaboration.

1. Model specification: include time fixed effects (or explicit seasonal dummies) to absorb calendar-driven confounding, a lagged spend term if the effect isn't instantaneous (media effects often build over 1-4 weeks), and controls for known concurrent drivers like price promotions and competitor actions where you have data. Where you don't have direct competitor data, a proxy (search interest, category-level sales) can partially substitute.
2. Omitted-variable bias tests: check whether adding a new plausible confounder materially changes the coefficient on spend (a large swing signals the earlier specification was incomplete); a formal Hausman-style comparison across specifications, or simply reporting the coefficient's sensitivity across a handful of reasonable specifications, both work as evidence of stability or fragility.
3. Robustness checks: re-estimate on subsamples (different regions, different time windows) and check the sign and rough magnitude hold; run a placebo test where you artificially assign a "fake" spend increase to a period where none happened and confirm the model doesn't manufacture a spurious effect there.

Worked example. A base model regresses weekly sales on weekly spend alone and finds a large, seemingly significant coefficient. Adding month fixed effects (to absorb seasonality) cuts the coefficient by half, revealing that a chunk of the naive correlation was seasonal co-movement, not a real spend effect. Further adding a lagged spend term (spend from two weeks prior) reveals part of the remaining "effect" is actually spend responding to sales momentum (marketing budgets often increase after a good sales week), not the reverse; this insight comes specifically from checking whether the coefficient on CURRENT spend shrinks once lagged spend and lagged sales are both in the model, which would suggest reverse causation or simultaneity rather than the marketing team's spend driving sales.

Trade-offs and pitfalls. A regression with lots of controls can look rigorous while still being wrong if it's missing an important confounder that was never measured (competitor promotional calendars, for instance); the honest response to that risk is to pair the regression with an experiment (a randomized geo-holdout) whenever the stakes justify the cost, rather than treating a well-specified regression as definitively causal on its own.

Direct answer. Off-policy (counterfactual) evaluation estimates how a NEW policy would have performed, using only data logged under an OLD policy, without deploying the new policy live; the main families of estimators are importance-weighting approaches (reweighting logged outcomes by how differently the new policy would have acted compared to the old one), direct/model-based approaches (predicting the new policy's outcome from a learned reward model), and doubly robust approaches (combining both, for the same robustness benefit as in the treatment-effect setting).

Structured elaboration.

• Importance-weighting family (inverse propensity scoring and its variants): reweight each logged (context, action, reward) triple by the ratio of the new policy's probability of taking that action to the old (logging) policy's probability of having taken it; this is unbiased under the right assumptions but can have very high variance when the two policies disagree substantially on which actions to take, since the ratio can become extreme. A self-normalized variant reduces this variance at the cost of a small bias, a common and often worthwhile trade in practice.
• Direct/model-based method: train a model predicting reward from context and action using the logged data, then use it to predict what the new policy's expected reward would be; lower variance than importance weighting, but biased if the reward model is misspecified, and it can look deceptively confident even when wrong.
• Doubly robust estimator: combines both, consistent if either the importance weights or the reward model is correctly specified, generally the preferred choice when feasible.

Where each can fail. Importance-weighting-based methods fail when the logging policy has essentially zero probability of ever taking an action the new policy strongly prefers (no support for that action in the logged data at all, an overlap violation exactly analogous to poor propensity-score overlap in the observational treatment-effect setting); the direct method fails silently when the reward model is systematically wrong in a way that isn't obvious from its training performance; and ALL of these methods degrade when non-stationarity means the logged data no longer reflects current user behavior (the world the old policy was logged in has meaningfully changed by the time you're evaluating the new one), since the entire framework assumes the logging data still represents the current population and context distribution.

Trade-offs and pitfalls. Off-policy evaluation is attractive because it avoids the cost and risk of a live deployment for the new policy, but it should be treated as a SCREENING step, ruling out clearly bad candidate policies cheaply, rather than a substitute for a final live validation before full rollout, precisely because the overlap and non-stationarity failure modes can silently produce a confidently wrong estimate that looks reasonable.

Direct answer. An instrumental variable (IV) is a variable that shifts treatment without having any other path to the outcome, letting you extract a causal effect even when unobserved confounding makes a direct comparison untrustworthy. For it to be valid, it needs three properties: it must actually move the treatment (relevance), it must affect the outcome only through the treatment and not through any other channel (the exclusion restriction), and it must not itself be driven by the same unobserved factors that drive the outcome (exchangeability, sometimes called as-if-random assignment of the instrument).

Structured elaboration and candidate instruments.

• Server-load-driven delay in sending a reminder email: the delay is caused by infrastructure load, plausibly unrelated to a specific user's purchase propensity (supports exchangeability), it clearly shifts whether/when a user receives the reminder (relevance), and as long as the delay itself has no other channel to affect purchases besides through receiving the email later (exclusion), it's a workable instrument. The risk to check: does high server load correlate with high traffic days that independently affect purchase behavior?
• A policy change that shifted eligibility as-if-randomly: for example, a threshold that changed for reasons unrelated to individual user behavior. Relevance is clear if eligibility strongly predicts uptake; exclusion requires that the policy change didn't also coincide with some other simultaneous change affecting the outcome.
• Geographic or time-based rollout staggering driven by operational constraints (which warehouse or region got a new feature first, for reasons like engineering capacity rather than user characteristics): relevance holds if the rollout order predicts exposure; exchangeability requires the rollout order wasn't itself chosen based on expected performance in that region.

For estimation, two-stage least squares (2SLS) regresses the treatment on the instrument (and any exogenous controls) in the first stage, then regresses the outcome on the predicted treatment from that first stage. The coefficient on predicted treatment is the IV estimate.

Worked example. Formally, with instrument $Z$, treatment $D$, and outcome $Y$:

$$\hat{D} = \hat{\pi}_0 + \hat{\pi}_1 Z \quad \text{(first stage)}$$ $$Y = \beta_0 + \beta_1 \hat{D} + \varepsilon \quad \text{(second stage)}$$

The first-stage F-statistic on $Z$ is the standard instrument-strength diagnostic; a common rule of thumb flags instruments weaker than roughly F=10 as unreliable, since a weak first stage amplifies any small violation of exclusion into a large bias in $\beta_1$.

Trade-offs and pitfalls. Exclusion is the assumption that's almost never directly testable, since it requires arguing there's no OTHER path from the instrument to the outcome, which is a claim about the world rather than something the data alone can confirm; the strongest IV answers name the specific alternate channel a skeptic would raise and explain why it's implausible, rather than asserting exclusion by fiat.