Responsible AI: Fairness, Bias, and Interpretability Questions

Direct answer

A disparity in a credit-scoring or recidivism model can come from three genuinely different places that call for three different fixes: the model itself adding disparity beyond what its legitimate inputs justify (a modeling defect), a historically-driven policy legacy baked into a feature that looks legitimate but encodes past exclusion, or a present-day socioeconomic confounder that reflects real, current inequality the model did not create. The diagnostic plan is to build a causal graph separating these three channels, collect the specific data each channel needs to be measured (not assumed), condition on the legitimate inputs to isolate the model's own residual contribution, and only then decide whether to intervene on a feature, intervene on policy, or replace the model, since replacing the model fixes only the first of the three.

Structured elaboration

The causal graph to draw. Put the protected attribute $A$ at the root, and trace three distinct paths to the outcome $Y$:

1. $A \to H \to \text{CH} \to Y$: historical policy legacy ($H$, e.g. redlining-era exclusion from credit access) shapes a feature the model uses (credit-history depth, $\text{CH}$), which the model then scores.
2. $A \to S \to Y$: current socioeconomic status ($S$, income, employment stability), a legitimate predictor of repayment capacity that also correlates with $A$ through present-day societal inequality, not through the model or a specific historical policy.
3. $A \to Y$ directly, bypassing every legitimate feature: the model's own use of an $A$-correlated proxy (zip code, a device or channel signal) that adds disparity beyond what $\text{CH}$ and $S$ already justify.
   $\text{CH}$ is also driven by true repayment reliability $R$, a legitimate, largely $A$-independent signal that must be included in the graph so the historical-policy path is not confused with genuine behavior.

Data needed for each path. Path 1 needs historical documentation independent of the model's own features: historical redlining maps, branch-density or credit-access records by geography and era, anything that lets you estimate how much of today's credit-history depth is explained by past policy rather than present behavior. Path 2 needs direct, current socioeconomic indicators (income, employment duration, asset holdings) collected at the individual level, not inferred from the same features the model already uses, so it can be conditioned on independently. Path 3 needs the model's own feature list and coefficients (or, for a black-box model, its residual behavior after controlling for every legitimate input), since the model-itself channel is defined as whatever disparity is left over once the legitimate paths are accounted for.

Which variables to condition on. To isolate the model-itself channel, condition on every LEGITIMATE input the model is supposed to be using (here, $\text{CH}$ and $S$) and measure what disparity remains in $Y$; anything left over after conditioning on the full legitimate feature set is either an unmeasured legitimate factor (a gap in the graph, worth investigating before concluding it is the model) or the model's own excess reliance on a proxy. To separate the historical-policy channel from the socioeconomic channel within the legitimate paths, condition on $S$ alone and see how much of $\text{CH}$'s remaining correlation with $A$ survives; what is left traces to $H$ (historical policy) rather than to current socioeconomic status, since the two operate through different variables in the graph. Do not condition on $\text{CH}$ when trying to isolate $H$'s effect on the FINAL disparity in $Y$, since $\text{CH}$ is the mediator carrying that very effect, and conditioning on a mediator when you want its total effect is a classic identification mistake (it would "explain away" exactly the pathway under investigation).

When to advise an intervention versus replacing the model. This should be a size-and-locus judgment, not a reflexive one:

• If the residual model-itself channel (disparity remaining after conditioning on all legitimate inputs) is a material share of the total, that is direct evidence the model is doing something beyond scoring its stated inputs, and the model needs to be retrained or re-specified (remove the offending proxy, or add an in-processing fairness constraint), because the defect is intrinsic to the model, not to its inputs.
• If most of the disparity traces to the historical-policy channel through a specific feature (credit-history depth encoding redlining-era exclusion), the fix is targeted at that FEATURE, not the whole model: substitute a less contaminated proxy for the same underlying construct (alternative credit-building data, rental/utility payment history), or explicitly adjust the feature for its historical-policy component before it enters any model, current or future.
• If most of the disparity traces to the socioeconomic channel and reflects a real, current difference in repayment capacity that predates and sits outside the model, the model may be statistically accurate and the disparity is a policy question above the model layer (whether and how to price risk that correlates with structural inequality, whether to offer alternative underwriting paths), not a defect to be engineered away inside this model. Advising "replace the model" here would not remove the disparity, since a different, equally accurate model trained on the same socioeconomic reality would show the same gap.
• In practice expect a mix of all three, in which case sequence the response: fix the model-itself channel first (it is usually the cheapest and most defensible fix), then the feature-level historical-policy contamination, and treat the socioeconomic-confounder residual as a business and policy decision to escalate, not silently absorb into "the model is fine."

Worked example

A credit-scoring SCM with all three channels present simultaneously, at a scale that makes each channel's share estimable with negligible noise:

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(9)
n = 300_000

A = rng.integers(0, 2, n).astype(float)
H = 0.70 * A + rng.normal(0, 1, n)              # historical policy legacy, correlated with A
S = 0.50 * A + rng.normal(0, 1, n)               # current socioeconomic status, correlated with A
R = -0.02 * A + rng.normal(0, 1, n)              # true repayment reliability: near-zero real gap
CH = 1.0 * R + 0.9 * H + rng.normal(0, 0.5, n)   # credit-history depth: true behavior + historical suppression

noise_model = rng.normal(0, 0.5, n)
Y_legit = 0.6 * CH + 0.8 * S + noise_model                    # a model using ONLY legitimate inputs
Y_actual = 0.6 * CH + 0.8 * S + 0.25 * A + noise_model         # the deployed model ALSO uses an A-correlated proxy

gap_actual = Y_actual[A == 1].mean() - Y_actual[A == 0].mean()
gap_legit = Y_legit[A == 1].mean() - Y_legit[A == 0].mean()
model_itself_share = gap_actual - gap_legit
print(f"observed disparity, deployed model: {gap_actual:.4f}")
print(f"disparity with only legitimate inputs (CH, S): {gap_legit:.4f}")
print(f"share of total disparity from the model itself: {model_itself_share/gap_actual*100:.1f}%")

coef_S = 0.8 * 0.50
coef_H = 0.6 * 0.9 * 0.70
coef_R = 0.6 * 1.0 * (-0.02)
implied_total = coef_S + coef_H + coef_R
print(f"share from current socioeconomic status (S): {coef_S/implied_total*100:.1f}%")
print(f"share from historical-policy-contaminated credit history (H via CH): {coef_H/implied_total*100:.1f}%")
print(f"share from true repayment-behavior gap (R via CH): {coef_R/implied_total*100:.1f}%")

X = np.column_stack([A, CH, S])
reg = LinearRegression().fit(X, Y_actual)
print(f"regression of Y_actual on [A, CH, S], recovered coefficient on A: {reg.coef_[0]:.4f}")

Executed output:

observed disparity, deployed model: 1.0150
disparity with only legitimate inputs (CH, S): 0.7650
share of total disparity from the model itself: 24.6%
share from current socioeconomic status (S): 52.2%
share from historical-policy-contaminated credit history (H via CH): 49.3%
share from true repayment-behavior gap (R via CH): -1.6%
regression of Y_actual on [A, CH, S], recovered coefficient on A: 0.2509

Of the total 1.0150-point disparity, 24.6% comes from the model's own excess reliance on an $A$-correlated proxy (confirmed by the regression recovering the model's true 0.25 proxy coefficient almost exactly, 0.2509, once $\text{CH}$ and $S$ are conditioned on). Of the remaining, legitimate-input-driven disparity, 52.2% comes from current socioeconomic status and 49.3% from the historical-policy-contaminated credit-history feature, while the true behavioral gap contributes essentially nothing (-1.6%, statistical noise around zero). This is exactly the mixed case: the model should be fixed first (retrain without the proxy, recovering roughly a quarter of the gap immediately), the credit-history feature needs a separate historical-policy adjustment, and the socioeconomic-status share should go to the business as a policy question rather than be treated as something a model change alone can or should erase.

Trade-offs and pitfalls

The most common wrong turn is collapsing all three channels into "model bias" and responding only by retraining or adding a fairness constraint; the worked example shows that would address only about a quarter of the disparity here and would leave the historical-policy and socioeconomic channels completely untouched, likely triggering a second audit finding shortly after the first is closed. A second pitfall is conditioning on the mediator ($\text{CH}$) when trying to measure the historical-policy channel's total effect on $Y$, which would statistically absorb the very effect under investigation and produce a false all-clear. A third is assuming a socioeconomic-confounder finding means "nothing to do here": even when the model is statistically accurate given real socioeconomic disparity, the business decision of whether to price that disparity into consumer credit outcomes is a legitimate and often legally scrutinized policy choice, and treating it as automatically out of scope for the audit is itself a judgment call that should be surfaced, not made silently. Finally, all of this depends on getting the graph right in the first place; an unmeasured legitimate factor missing from the graph will masquerade as "model itself" disparity in the conditioning step, so any material model-itself share should trigger a check for missing legitimate variables before concluding the model needs to be replaced.

Direct answer

Counterfactual fairness (Kusner et al., 2017) asks a per-individual, not per-population, question: for this specific person, would the outcome have been different if only their sensitive attribute $A$ had been different, holding everything about them that is causally independent of $A$ fixed? The check has three structural steps against a structural causal model (SCM) over $(A, X, Y)$: abduction (infer each individual's exogenous noise from what was observed), action (intervene to flip $A$), and prediction (recompute the downstream variables under the flipped world using the same noise). A predictor is counterfactually fair if its output does not change for any individual under this flip; the estimation step has to compare factual and counterfactual outcomes PAIRED per individual, because a population-level distributional comparison can miss the violation entirely when the flip is symmetric across the population.

Structured elaboration

The SCM. Let $A$ be the sensitive attribute (a root/exogenous variable, e.g. gender), $X$ a mediator that $A$ causally influences (e.g. years of experience shaped by unequal historical opportunity), and $Y$ the outcome (e.g. a hiring score) causally downstream of both:

where $U_A, U_X, U_Y$ are mutually independent exogenous noise terms and $f_X, f_Y$ are the structural equations. $A$'s effect on $Y$ can travel two ways: directly ($A \to Y$) and indirectly through the mediator ($A \to X \to Y$); both are potential sources of unfairness the check needs to catch.

Pseudocode, the three-step abduction/action/prediction procedure applied per individual $i$:

for each individual i with observed (a_i, x_i, y_i):

    # 1. ABDUCTION: infer the exogenous noise this individual actually has,
    #    given the SCM's structural equations and what was observed.
    #    (exact if f_X, f_Y are invertible in U; posterior sampling / MCMC
    #     if they are not, e.g. non-invertible or the SCM has unobserved confounders)
    u_x_i = invert_f_X(a_i, x_i)
    u_y_i = invert_f_Y(a_i, x_i, y_i)          # only needed if testing Y itself,
                                                 # not needed if testing a fitted g(A, X)

    # 2. ACTION: intervene, do(A := 1 - a_i), the counterfactual flip.
    a_cf_i = 1 - a_i

    # 3. PREDICTION: recompute downstream variables under the SAME noise,
    #    now with the counterfactual A.
    x_cf_i = f_X(a_cf_i, u_x_i)
    y_cf_i = f_Y(a_cf_i, x_cf_i, u_y_i)         # or g(a_cf_i, x_cf_i) for a fitted predictor g

    store (y_i_or_ghat_i, y_cf_i_or_ghat_cf_i)

Sampling steps (building the dataset the check runs against, distinct from the per-individual counterfactual computation above): draw $U_A, U_X, U_Y$ independently from their assumed distributions, push them forward through $f_X$ then $f_Y$ to generate a synthetic observational sample $(A, X, Y)$ of $n$ individuals; on real data, skip this and abduct directly from the observed rows instead of simulating them. When $f_X$ or $f_Y$ is not exactly invertible (nonlinear, or the SCM includes latent confounders), abduction becomes approximate: sample $U$ from its posterior given the observed variables (for example, via Monte Carlo / MCMC, or a variational approximation), and repeat the action-and-prediction steps for several posterior draws to get a distribution of counterfactual outcomes per individual rather than a single point value.

Estimation steps: compute, for every individual, the counterfactual shift $\Delta_i = \hat{y}^{cf}_i - \hat{y}_i$. Report:

1. The mean and standard deviation of $\Delta_i$ across individuals (an aggregate size-of-violation summary).
2. The fraction of individuals whose $|\Delta_i|$ exceeds a materiality threshold (a practical, business-legible "how many people would actually see a different outcome" number).
3. A paired test (for example, a paired t-test or Wilcoxon signed-rank test on $\Delta_i$, or simply checking whether $\Delta_i$ is materially different from 0 for most individuals), NOT an unpaired two-sample test comparing the pooled factual distribution to the pooled counterfactual distribution. This distinction matters: if the flip sends roughly half the population from $A=0$ to $A=1$ and the other half from $A=1$ to $A=0$ with a symmetric effect, the pooled factual and counterfactual distributions can look statistically indistinguishable in aggregate even though every single individual's own outcome changed substantially. The question asks for invariance "for individuals," and only a paired, per-individual comparison actually tests that.

Worked example

import numpy as np
from sklearn.linear_model import LinearRegression
from scipy import stats

rng = np.random.default_rng(42)
n = 5000

U_A = rng.integers(0, 2, n)
A = U_A.astype(float)
U_X = rng.normal(0, 1, n)
X = 2.0 * A + U_X                      # A causally shifts the mediator X
U_Y = rng.normal(0, 1, n)
Y = 3.0 * X + 1.5 * A + U_Y            # A has BOTH a direct and an X-mediated effect on Y

naive_model = LinearRegression().fit(np.column_stack([A, X]), Y)   # uses A directly: unfair by construction
fair_model = LinearRegression().fit(U_X.reshape(-1, 1), Y)          # uses only the A-independent residual

def counterfactual_check(model, use_features):
    U_X_hat = X - 2.0 * A          # abduction: exact here since f_X is linear and invertible in U_X
    A_cf = 1.0 - A                 # action: do(A := 1-A)
    X_cf = 2.0 * A_cf + U_X_hat    # prediction: recompute the mediator under the flip
    if use_features == "AX":
        return model.predict(np.column_stack([A, X])), model.predict(np.column_stack([A_cf, X_cf]))
    return model.predict(U_X_hat.reshape(-1, 1)), model.predict(U_X_hat.reshape(-1, 1))

pred_f_naive, pred_cf_naive = counterfactual_check(naive_model, "AX")
pred_f_fair, pred_cf_fair = counterfactual_check(fair_model, "UX")

delta_naive = pred_cf_naive - pred_f_naive
delta_fair = pred_cf_fair - pred_f_fair
print(f"naive: mean shift={delta_naive.mean():.4f}, frac |shift|>0.5={np.mean(np.abs(delta_naive)>0.5):.4f}")
print(f"fair:  mean shift={delta_fair.mean():.6f}, frac |shift|>0.5={np.mean(np.abs(delta_fair)>0.5):.4f}")

ks_naive = stats.ks_2samp(pred_f_naive, pred_cf_naive)
print(f"naive UNPAIRED (pooled) KS statistic: {ks_naive.statistic:.4f}, p={ks_naive.pvalue:.2e}")

Executed output:

naive: mean shift=0.0898, frac |shift|>0.5=1.0000
fair:  mean shift=0.000000, frac |shift|>0.5=0.0000
naive UNPAIRED (pooled) KS statistic: 0.0234, p=1.29e-01

The naive model, which conditions on $A$ directly, shifts every single individual's ($100\%$) prediction by more than 0.5 when their $A$ is counterfactually flipped, clearly violating counterfactual fairness. The fair model, trained only on the noise term $U_X$ that abduction shows is independent of $A$, shows exactly zero shift for every individual, by construction. But the naive model's pooled (unpaired) factual-versus-counterfactual KS test comes back statistically non-significant (p = 0.129): the population-level distributions look similar because the flip is symmetric (roughly half the population moves up, half moves down, by comparable magnitudes), even though the paired, per-individual check shows every individual is affected. This is exactly the gap the estimation step above is written to avoid.

Trade-offs and pitfalls

The most common wrong turn is running the pooled, unpaired comparison shown above and concluding the model is fair because the aggregate distributions match; the worked example shows this passes even for a model that changes literally every individual's prediction. A second pitfall is assuming abduction is exact: it only inverts cleanly when $f_X$ and $f_Y$ are invertible functions of their noise terms, which almost never holds for a nonlinear model or a neural-network-based structural equation, so real applications need approximate (posterior-sampling) abduction and should report a distribution of counterfactual outcomes per individual, not a single point estimate. Third, the entire exercise inherits the correctness of the assumed causal graph: if the true graph has an unmodeled confounder between $A$ and $X$, or the mediator is mislabeled as legitimate when it is itself downstream of historical discrimination, the counterfactual-fairness check will faithfully certify the wrong graph as fair. Finally, dropping $A$-influenced features entirely (as the "fair model" does here) achieves counterfactual invariance by construction, but at the cost of discarding any legitimate predictive signal those features also carried, so in practice the harder and more common task is separating the part of a mediator's variance that is a legitimate signal from the part that is a proxy for historical discrimination, not simply excluding the mediator outright.

Direct answer

Measure fairness at the INTERSECTION of gender and race, not just each dimension separately, since a model can look acceptable on each marginal axis while still failing badly for a specific combination (a real, well-documented pattern, not a hypothetical). Where gender or race is missing, use a soft, probabilistic group-membership estimate rather than dropping those applicants or guessing a hard label, and carry that uncertainty through the fairness calculation itself, including a minimum EFFECTIVE sample size per intersectional cell. Mitigate at two levels together: reweigh training examples toward the group-conditional base rate, AND directly address any resume feature that is itself a proxy for a protected attribute (a referral flag standing in for network access, an employment-gap flag standing in for historical caregiving norms), because reweighing alone will not undo a disparity that is being driven through a specific, still-present proxy feature. Govern this with a standing audit cadence, not a one-time check, since both the applicant pool and the model drift over time.

Structured elaboration

Intersectional fairness metrics. For $K$ protected dimensions with 2 categories each, there are $2^K$ intersectional cells (2 dimensions here: gender x race, 4 cells), and the fairness metric of interest, a disparate-impact-style selection-rate ratio here, has to be computed for EACH cell against a reference (the cell with the highest observed rate, following the same 80 percent rule convention used for single-attribute analysis), not just for gender marginally and race marginally. This matters because marginal fairness is a necessary but not sufficient condition for intersectional fairness: a model can show acceptable gender-only and race-only gaps while its worst-affected group is a specific combination that neither marginal view would surface. Because intersectional cells are smaller by construction (splitting a population four ways instead of two), the SAME minimum-sample-size discipline used for single-attribute audits applies here with more force, and testing many cells at once also raises a real multiple-comparisons concern that a single-attribute test does not face as acutely.

Handling partially missing protected attributes. Applicants who decline to self-report gender or race should not be silently excluded (which biases the audited population toward whoever chose to disclose) or silently assigned a hard guessed label (which fabricates certainty that does not exist). The standard approach is a soft, probabilistic group-membership estimate, conceptually similar to Bayesian Improved Surname Geocoding used in fair-lending analysis, that assigns each applicant a probability distribution over the possible cells rather than a single hard label. Every downstream fairness computation should then use those probabilities as WEIGHTS (an applicant with missing data contributes a FRACTION of a person to each of several cells) rather than collapsing to a point estimate, and the resulting EFFECTIVE sample size per cell (the sum of weights, not a raw row count) is what the minimum-sample-size gate should actually check.

Historical selection bias. A hiring model trained on past hiring OUTCOMES inherits every bias that shaped those outcomes, and two mechanisms are common and distinct: a feature that gives a real but partly UNEARNED boost (a referral flag, where network effects are real but access to the network is itself unevenly distributed across groups for historical reasons unrelated to merit), and a feature that imposes an UNEARNED penalty (an employment-gap flag, where the gap itself may correlate with historical caregiving norms rather than any real capability difference). Neither of these requires the model to see gender or race directly; both function as PROXY features that reintroduce the disparity through the back door, which is exactly why "we don't use the protected attribute as a model input" does not, on its own, guarantee a fair outcome.

Mitigation strategy: two levels, not one. Example-level reweighing (in the style of Kamiran and Calders, adapted here to soft group membership) adjusts each training example's WEIGHT so that, in expectation, each group's outcome rate is pulled toward the overall base rate; this is necessary but, as the worked example below shows directly, it is not sufficient when a specific feature is a strong, still-present proxy for the disparity. Feature-level intervention, auditing which resume features function as proxies and adjusting or removing the ones without a defensible, job-relevant justification (a referral flag is the clearer case here; an employment-gap flag needs more care, since it can also legitimately correlate with a real skills gap and should be handled with a nuanced policy, not a blanket removal), addresses the mechanism reweighing alone cannot reach.

Auditing and governance for ongoing monitoring. A one-time audit is not sufficient because both the applicant pool and the model retrain over time. A durable governance program needs: a fixed re-audit cadence (not just "when someone asks"), drift monitoring on both the model's intersectional selection rates AND the composition of the applicant pool itself (a shift in who applies can move a fairness metric even with an unchanged model), a documented fairness impact assessment updated at every model retrain and reviewed by a named accountable owner, a human-review or appeal channel for rejected candidates, and a pre-agreed rollback trigger and procedure if a monitoring alert fires, so a response plan exists before it is needed under pressure.

Worked example

A synthetic hiring dataset with a real proxy-driven historical bias (referral favoring one intersectional cell, an employment-gap flag disproportionately affecting another), 12 percent of applicants missing self-reported gender and race (handled with soft, Dirichlet-perturbed group-membership probabilities standing in for a real name/geography-based estimate), evaluated across all four intersectional cells under three progressively stronger interventions:

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(6)
n = 3000

# ---- Protected attributes: gender in {M, F} x race in {A, B} -> 4 intersectional
# cells. 12% of applicants have NO self-reported gender/race (a realistic
# non-disclosure rate). ----
gender = rng.choice(["M", "F"], size=n, p=[0.55, 0.45])
race = rng.choice(["A", "B"], size=n, p=[0.65, 0.35])
cells = np.array([f"{g}-{r}" for g, r in zip(gender, race)])
CELL_NAMES = ["M-A", "M-B", "F-A", "F-B"]
missing_mask = rng.uniform(0, 1, n) < 0.12

# For applicants with MISSING self-report, simulate a soft, IMPERFECT proxy-based
# group-membership estimate (standing in for a real technique like Bayesian
# Improved Surname Geocoding): a probability vector over the 4 cells, centered on
# the true cell but with meaningful estimation noise, never a hard guess.
def soft_membership(true_cell_idx, noise_conc=6.0, r=rng):
    alpha = np.ones(4) * 0.5
    alpha[true_cell_idx] += noise_conc
    return r.dirichlet(alpha)

cell_idx_true = np.array([CELL_NAMES.index(c) for c in cells])
membership_prob = np.zeros((n, 4))
for i in range(n):
    if missing_mask[i]:
        membership_prob[i] = soft_membership(cell_idx_true[i])
    else:
        membership_prob[i, cell_idx_true[i]] = 1.0   # observed: hard, certain membership

# ---- Resume features. has_gap is a PROXY correlated with gender (F), reflecting
# historical caregiving-driven career gaps; referral is a PROXY correlated with
# the historically dominant network (M-A), reflecting HISTORICAL SELECTION BIAS
# baked into who gets referred in the first place. ----
years_experience = rng.normal(6, 3, n).clip(0, None)
prior_title_seniority = rng.normal(3, 1.2, n).clip(0, None)
has_gap = (rng.uniform(0, 1, n) < np.where(gender == "F", 0.30, 0.10)).astype(float)
referral = (rng.uniform(0, 1, n) < np.where(cells == "M-A", 0.45, 0.15)).astype(float)
true_quality = 0.4 * years_experience + 0.6 * prior_title_seniority + rng.normal(0, 1.5, n)

# ---- HISTORICAL hiring label: the data the current model would be built on.
# Referral gives a real boost (network effects are real) but ALSO an unjustified
# boost beyond what quality alone predicts; a gap imposes an unjustified penalty
# beyond quality alone. Both are the historical selection bias baked into the
# label this exercise has to contend with. ----
hist_logit = (0.35 * true_quality - 3.0) + 1.8 * referral - 1.4 * has_gap
hist_hire = (rng.uniform(0, 1, n) < 1 / (1 + np.exp(-hist_logit))).astype(int)

X_full = np.column_stack([years_experience, prior_title_seniority, has_gap, referral])
X_no_referral = np.column_stack([years_experience, prior_title_seniority, has_gap])

def intersectional_rates(reco, membership_prob, min_effective_n=50):
    report = {}
    for k, name in enumerate(CELL_NAMES):
        w = membership_prob[:, k]
        eff_n = w.sum()
        rate = np.sum(w * reco) / eff_n if eff_n > 0 else float("nan")
        report[name] = {"effective_n": round(eff_n, 1), "reco_rate": round(rate, 4)}
    max_rate = max(v["reco_rate"] for v in report.values() if v["effective_n"] >= min_effective_n)
    for name, v in report.items():
        if v["effective_n"] < min_effective_n:
            v["disparate_impact_ratio"], v["flag"] = None, "INSUFFICIENT_EFFECTIVE_SAMPLE"
        else:
            v["disparate_impact_ratio"] = round(v["reco_rate"] / max_rate, 4)
            v["flag"] = "FLAGGED" if v["disparate_impact_ratio"] < 0.8 else "OK"
    return report

def threshold_for_rate(scores, target_rate):
    return np.quantile(scores, 1 - target_rate)

# ---- Reweighing (Kamiran & Calders style), extended to SOFT group membership:
# each cell's expected (soft-weighted) hire rate is pulled toward the overall
# base rate via a per-cell factor, blended per row by its membership probabilities. ----
overall_hire_rate = hist_hire.mean()
cell_reweight_factor = np.zeros(4)
for k in range(4):
    w = membership_prob[:, k]
    cell_hire_rate = np.sum(w * hist_hire) / w.sum()
    cell_reweight_factor[k] = overall_hire_rate / cell_hire_rate if cell_hire_rate > 0 else 1.0
sample_weight = membership_prob @ cell_reweight_factor

print("Per-cell reweighing factors (soft-membership-weighted):")
for name, factor in zip(CELL_NAMES, cell_reweight_factor):
    print(f"  {name}: {factor:.4f}")

TARGET_RATE = 0.15  # a fixed overall selection rate, so all three models are
                     # compared at the SAME hiring volume, not a fixed score cutoff
configs = [
    ("BASELINE (all features, no reweighing)", X_full, None),
    ("REWEIGHED (all features)", X_full, sample_weight),
    ("REWEIGHED + referral proxy feature removed", X_no_referral, sample_weight),
]
for label, Xv, sw in configs:
    model = LogisticRegression(max_iter=1000).fit(Xv, hist_hire, sample_weight=sw)
    scores = model.predict_proba(Xv)[:, 1]
    t = threshold_for_rate(scores, TARGET_RATE)
    reco = (scores >= t).astype(int)
    print(f"\n=== {label} (overall selection rate={reco.mean():.4f}) ===")
    for name, v in intersectional_rates(reco, membership_prob).items():
        print(f"  {name}: {v}")

Executed output:

Per-cell reweighing factors (soft-membership-weighted):
  M-A: 0.7426
  M-B: 1.1065
  F-A: 1.3238
  F-B: 1.2507

=== BASELINE (all features, no reweighing) (overall selection rate=0.1500) ===
  M-A: {'effective_n': np.float64(1061.7), 'reco_rate': np.float64(0.2679), 'disparate_impact_ratio': np.float64(1.0), 'flag': 'OK'}
  M-B: {'effective_n': np.float64(574.6), 'reco_rate': np.float64(0.1075), 'disparate_impact_ratio': np.float64(0.4013), 'flag': 'FLAGGED'}
  F-A: {'effective_n': np.float64(893.8), 'reco_rate': np.float64(0.0698), 'disparate_impact_ratio': np.float64(0.2605), 'flag': 'FLAGGED'}
  F-B: {'effective_n': np.float64(469.8), 'reco_rate': np.float64(0.0882), 'disparate_impact_ratio': np.float64(0.3292), 'flag': 'FLAGGED'}

=== REWEIGHED (all features) (overall selection rate=0.1500) ===
  M-A: {'effective_n': np.float64(1061.7), 'reco_rate': np.float64(0.2695), 'disparate_impact_ratio': np.float64(1.0), 'flag': 'OK'}
  M-B: {'effective_n': np.float64(574.6), 'reco_rate': np.float64(0.108), 'disparate_impact_ratio': np.float64(0.4007), 'flag': 'FLAGGED'}
  F-A: {'effective_n': np.float64(893.8), 'reco_rate': np.float64(0.0687), 'disparate_impact_ratio': np.float64(0.2549), 'flag': 'FLAGGED'}
  F-B: {'effective_n': np.float64(469.8), 'reco_rate': np.float64(0.0861), 'disparate_impact_ratio': np.float64(0.3195), 'flag': 'FLAGGED'}

=== REWEIGHED + referral proxy feature removed (overall selection rate=0.1500) ===
  M-A: {'effective_n': np.float64(1061.7), 'reco_rate': np.float64(0.1687), 'disparate_impact_ratio': np.float64(1.0), 'flag': 'OK'}
  M-B: {'effective_n': np.float64(574.6), 'reco_rate': np.float64(0.1642), 'disparate_impact_ratio': np.float64(0.9733), 'flag': 'OK'}
  F-A: {'effective_n': np.float64(893.8), 'reco_rate': np.float64(0.1375), 'disparate_impact_ratio': np.float64(0.8151), 'flag': 'OK'}
  F-B: {'effective_n': np.float64(469.8), 'reco_rate': np.float64(0.1143), 'disparate_impact_ratio': np.float64(0.6775), 'flag': 'FLAGGED'}

This is the two-level mitigation point made concretely, not asserted. The baseline model, at a fixed 15 percent overall selection rate so every comparison is at the same hiring volume, shows M-A (the historically networked, non-gap-affected cell) far ahead of every other cell, with M-B, F-A, and F-B all clearing FLAGGED territory (ratios of 0.40, 0.26, and 0.33). Reweighing alone barely moves any of these numbers (0.40, 0.25, 0.32), confirming that example-level reweighing cannot undo a disparity that is still being actively driven through a live proxy feature (referral) present in the model's inputs; the model keeps relying on that feature regardless of how the training examples are weighted. Only after ALSO removing the referral proxy feature, on top of reweighing, do M-B (0.9733) and F-A (0.8151) clear the fairness bar, which shows the feature-level intervention was doing the real work the reweighing alone could not. Critically, F-B, the doubly-disadvantaged intersectional cell, remains FLAGGED (0.6775) even after both interventions, which is exactly the finding an intersectional-only-after-marginal-fixes analysis would miss: fixing what looks like the two separate axes of disparity did not fix their intersection, because F-B's disadvantage compounds effects from both the removed referral proxy AND the still-present employment-gap penalty in a way neither single-axis fix fully addresses.

Trade-offs and pitfalls

The clearest pitfall, demonstrated directly above, is declaring victory after fixing what marginal (gender-only, race-only) metrics show, without checking the actual intersectional cells; F-B here would have looked adequately addressed by any analysis that only checked gender and race separately after the two interventions, since both individual axes improved substantially. A second is treating reweighing as a universal fix; it corrects for group-conditional LABEL imbalance but does nothing about a specific feature that is itself a strong, still-present channel for the disparity, and the worked example shows this is not a subtle effect, reweighing alone left every ratio essentially unchanged. A third is over-aggressively removing the employment-gap feature the same way the referral feature was removed here; unlike referral, a gap flag CAN carry some legitimate, job-relevant signal in specific contexts, so a blanket removal policy needs more nuance and its own justification, not a copy-paste of whatever worked for the clearer proxy. Finally, treating the soft group-membership estimate for applicants with missing self-report as if it were ground truth is a risk in the opposite direction: the effective sample sizes above (469.8 to 1061.7) already reflect the dilution from imperfect imputation, and any fairness conclusion drawn from a cell whose effective sample size is small or whose imputation confidence is low should be reported with that uncertainty attached, not presented with the same confidence as a cell built entirely from self-reported, observed data.

Direct answer

SHAP (SHapley Additive exPlanations) and similar feature-attribution methods answer "how much did this feature's value contribute to this model's prediction," which is a statement about the model's learned associations, not "what would happen to the outcome if we intervened on this feature," which is a causal statement. The two coincide only under specific assumptions that almost never hold by default (no unmeasured confounding, a correctly specified causal graph, adequate support/overlap in the data), so moving from a SHAP-style attribution to a causal claim requires an explicit identification strategy on top of the attribution, and the two kinds of claim must be labeled differently to stakeholders rather than presented as interchangeable numbers.

Structured elaboration

Why SHAP is correlational by default. SHAP computes a feature's average marginal contribution to the model's output across coalitions of other features, using either a conditional value function (which can attribute credit to a feature purely because it is correlated with another feature that the model actually uses) or an interventional/marginal value function (which breaks that specific correlation-with-other-features issue, but still measures the MODEL's learned association with the observational data it was trained on, not the true data-generating causal effect). If the training data contains an unmeasured confounder driving both a feature and the outcome, the model can and typically will pick up that spurious correlation, and SHAP will faithfully report it as an important contribution, because SHAP is explaining the model, not the world.

Required assumptions to move from attribution to causal interpretation. All of the following must hold, and each is a claim that needs to be argued for, not assumed:

1. No unmeasured confounding (ignorability/exchangeability) between the feature of interest and the outcome, given the variables you condition on: every common cause of the feature and the outcome must be measured and available to adjust for.
2. A correctly specified causal graph, at least a partial one that identifies which variables are confounders (common causes), mediators (variables that sit ON the causal path between the feature and the outcome, and should NOT be adjusted for, since doing so removes part of the feature's real effect), or colliders (variables CAUSED by both the feature and the outcome, where conditioning on them can create a spurious association between the two that did not exist before), relative to the feature-outcome relationship, since the correct adjustment set depends entirely on getting this structure right.
3. Positivity/overlap: the feature of interest must vary at every combination of the adjustment-set variables observed in the data; if a feature's value is nearly deterministic given the confounders, there is no data-driven way to separate the feature's own effect from the confounder's.
4. No interference between units (SUTVA): one individual's feature values and outcome are not causally affected by another individual's treatment status, which is usually reasonable for i.i.d. tabular data but not for networked or marketplace settings.
5. The model form does not introduce its own attenuation or amplification relative to the true structural relationship; a flexible model like a tree ensemble can compress or reshape an effect's apparent magnitude even when the direction is right, so the SIZE of a causally-validated attribution should be treated with more caution than its sign.

Identification strategy. Given a hypothesized causal graph:

1. Use the backdoor criterion to find an adjustment set: a set of variables that, once conditioned on, blocks every non-causal (confounding) path between the feature of interest and the outcome without blocking the feature's own causal path.
2. Estimate the feature's causal effect via regression adjustment, matching, inverse-propensity weighting, or a doubly-robust combination of the two, conditioning on the backdoor adjustment set, not on the model's other features indiscriminately (conditioning on a mediator of the feature's own effect would wrongly remove part of that effect, not just remove confounding).
3. If a valid backdoor adjustment set is not available (an unmeasured confounder is suspected), look for an instrumental variable (a variable that affects the feature but has no direct effect on the outcome except through the feature) or a natural experiment/quasi-random source of variation in the feature.
4. Compare the causal estimate to the attribution method's output as a diagnostic, not as a replacement: a large gap between a feature's SHAP importance and its backdoor-adjusted causal effect is itself a useful signal that the feature's apparent importance is substantially confounded, even before you fully trust either number's exact magnitude.

Presenting causal versus correlational claims honestly to stakeholders. Never let a single number labeled "importance" stand in for both. Use two clearly separated columns or sections: one labeled explicitly as associative importance (what SHAP or permutation importance measures, "the model relies on this feature this much"), and one labeled estimated causal effect, under stated assumptions (what the backdoor-adjusted or instrumental-variable estimate measures, "changing this feature is estimated to change the outcome by this much, ASSUMING no unmeasured confounding"). State the assumptions plainly next to the causal number, not in a footnote, and flag any feature where the two numbers disagree sharply as a feature that needs more investigation before anyone acts on it as if it were a lever to pull.

Worked example

A confounder $Z$ (e.g. overall customer engagement) drives both a feature $X_1$ (app opens) that has ZERO true causal effect on the outcome, and the outcome itself; a second feature $X_2$ (support-ticket response time) has a genuine causal effect:

import numpy as np
import shap
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(13)
n = 6000

Z = rng.normal(0, 1, n)
X1 = 1.2 * Z + rng.normal(0, 0.5, n)             # X1: purely a downstream effect of Z, NO causal effect on Y
X2 = rng.normal(0, 1, n)                          # X2: has a genuine causal effect on Y
Y = 2.0 * Z + 1.5 * X2 + rng.normal(0, 0.5, n)    # true structural equation: X1's causal weight is exactly 0

model = RandomForestRegressor(n_estimators=300, max_depth=6, random_state=0).fit(
    np.column_stack([X1, X2]), Y)

background = shap.sample(np.column_stack([X1, X2]), 200, random_state=0)
explainer = shap.TreeExplainer(model, background, feature_perturbation="interventional")
shap_values = explainer.shap_values(np.column_stack([X1, X2])[:1000])
mean_abs_shap_X1 = np.abs(shap_values[:, 0]).mean()
mean_abs_shap_X2 = np.abs(shap_values[:, 1]).mean()
print(f"mean |SHAP| for X1 (no true causal effect): {mean_abs_shap_X1:.4f}")
print(f"mean |SHAP| for X2 (true causal effect = 1.5): {mean_abs_shap_X2:.4f}")

reg_naive = LinearRegression().fit(np.column_stack([X1, X2]), Y)
reg_adjusted = LinearRegression().fit(np.column_stack([X1, X2, Z]), Y)  # backdoor adjustment: condition on Z
print(f"naive coefficient on X1 (not adjusted for Z): {reg_naive.coef_[0]:.4f}")
print(f"backdoor-adjusted coefficient on X1: {reg_adjusted.coef_[0]:.4f}")
print(f"backdoor-adjusted coefficient on X2: {reg_adjusted.coef_[1]:.4f}")

Executed output:

mean |SHAP| for X1 (no true causal effect): 1.4758
mean |SHAP| for X2 (true causal effect = 1.5): 1.1617
naive coefficient on X1 (not adjusted for Z): 1.4347
backdoor-adjusted coefficient on X1: -0.0005
backdoor-adjusted coefficient on X2: 1.4997

SHAP assigns $X_1$ a mean absolute contribution of 1.4758, essentially as large as $X_2$'s (1.1617), and the naive (unadjusted) regression coefficient on $X_1$ (1.4347) closely matches SHAP's magnitude, both driven by the same confounding through $Z$. Once $Z$ is adjusted for via the backdoor criterion, $X_1$'s coefficient collapses to -0.0005, correctly revealing its true causal effect of zero, while $X_2$'s adjusted coefficient (1.4997) matches its true generating coefficient (1.5) almost exactly. Reporting SHAP's 1.4758 for $X_1$ as if it meant "intervening on app opens would move the outcome by this much" would have been entirely wrong; the backdoor-adjusted estimate is what actually answers that question. Note also that $X_2$'s SHAP value (1.1617) is noticeably smaller than its true causal effect (1.4997), a reminder that even a correctly-signed attribution from a flexible model like a random forest can understate a real effect's magnitude, a separate imprecision from the confounding problem that affected $X_1$.

Trade-offs and pitfalls

The most damaging wrong turn is treating SHAP's ranking as a ranking of "what to intervene on to change the outcome," which the worked example shows can be actively backwards: the feature with the largest attribution here has zero causal effect. A second pitfall is assuming an interventional (marginal) SHAP value function alone solves the confounding problem; it removes one specific bias (crediting a feature for its correlation with OTHER MODEL FEATURES), but it does nothing about confounding in the underlying data-generating process, which is what the worked example demonstrates. Third, the entire causal identification step is only as good as the assumed graph: get the confounder set wrong (miss $Z$, or wrongly treat a mediator as a confounder and adjust for it) and the "causal" estimate is just as unreliable as the attribution it was meant to correct, so this approach earns its causal label only when the graph itself has been argued for with domain knowledge or external evidence, not asserted for convenience. Finally, resist compressing both numbers into one dashboard column under a single "importance" header for the sake of a cleaner report; the entire value of this exercise for stakeholders is the visible disagreement between the two numbers when it exists, and collapsing them erases exactly the signal that matters most.

A strong answer opens by naming the three definitions and stating plainly that they generally cannot all hold at once when base rates differ across groups.

Structured elaboration

Loan example. Say a bank approves loans with a risk score.

• Demographic parity means the same fraction of applicants in each demographic group gets approved, even if the groups have different true default rates.
• Equalized odds means that among applicants who would actually repay, the approval rate is the same across groups (equal TPR), and among those who would default, the rejection rate is the same across groups (equal FPR).
• Calibration means that a 0.2 default-risk score means a genuine 20% default probability whether the applicant is in group A or group B.

Worked example. If group A has a true default rate of 10% and group B has a true default rate of 30%, a single calibrated score function will naturally assign more high scores to group B. Forcing demographic parity on top of that calibration would require either denying good group-A applicants or approving bad group-B applicants purely to match rates, which breaks calibration. This is not a hypothetical: it is the mathematical content of the impossibility result once you fix differing base rates.

Trade-offs and pitfalls. (a) A regulator asking for "equal treatment across groups" usually means demographic parity or equalized odds, not calibration, so lean there. (b) A downstream risk-scoring use case (setting an interest rate, sizing a reserve) needs calibration, because a wrongly-calibrated score misprices risk for an entire group even if selection rates look fair. (c) The most common mistake is treating these three as compatible variations on "fairness" rather than as genuinely conflicting design choices; picking one is a policy decision, not a purely technical one, and should be made with legal and business stakeholders, not unilaterally by the model team.