Summary: Propensity score matching (PSM) estimates the causal effect of a binary “treatment” (feature rollout) by pairing treated and control units with similar probabilities of treatment given covariates, thereby reducing confounding from observed covariates.Assumptions:- Conditional ignorability (unconfoundedness): given covariates X, treatment is independent of potential outcomes (no unobserved confounders).- Positivity / overlap: 0 < P(T=1 | X) < 1 for all X (common support).- Correct specification of propensity model (or use flexible methods).Implementation steps:1. Define covariates X that predict both treatment assignment and outcome.2. Estimate propensity scores p(X) using logistic regression, random forest, or gradient boosting.3. Match treated to control (nearest‑neighbor, caliper, 1:k, with/without replacement, or optimal).4. Check balance and overlap; adjust matching parameters if needed.5. Estimate average treatment effect on treated (ATT) by comparing outcomes within matched pairs (simple difference, paired regression, or weighted estimator).6. Compute robust SEs (bootstrap or analytic formulas accounting for matching).Minimal Python example:python
# estimate propensity
from sklearn.linear_model import LogisticRegression
import numpy as np
lr = LogisticRegression(max_iter=1000)
lr.fit(X, treat)
ps = lr.predict_proba(X)[:,1]
# nearest neighbor matching (1:1, caliper optional)
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(n_neighbors=1)
nn.fit(ps[control_idx].reshape(-1,1))
dist, idx = nn.kneighbors(ps[treated_idx].reshape(-1,1))
Diagnostics to check:- Standardized Mean Differences (SMD) for each covariate before/after matching; aim SMD < 0.1 (or <0.05) after matching.- Variance ratios (close to 1).- Propensity score overlap: density/histogram or KS test; ensure common support and no extreme extrapolation.- Love plot (SMDs visualized), QQ plots for continuous covariates.- Check matched sample size and caliper exclusion rate.Practical pitfalls:- Unobserved confounding: PSM can't fix hidden biases—use domain expertise, sensitivity analyses (Rosenbaum bounds).- Poor overlap: many treated/controls unmatched; estimates extrapolate—consider trimming or restricting common support.- Model misspecification: use flexible models (GBM) or ensemble; but avoid overfitting.- Loss of sample size and precision with strict matching.- Incorrect variance estimation if matching ignored in SEs.Alternatives / complements:- Propensity score weighting (IPTW) to use more data; stabilized weights to reduce variance.- Doubly robust estimators (AIPW): combine outcome model + propensity for robustness.- Regression adjustment, covariate balancing propensity score (CBPS), entropy balancing.- Instrumental variables, difference-in-differences, or synthetic control when ignorability is implausible.Practical tips:- Pre-register covariates and analysis choices to avoid p-hacking.- Report balance tables, overlap plots, number matched, and sensitivity analyses to build stakeholder trust.