Responsible AI: Fairness, Bias, and Interpretability Questions

Direct answer: a strong STAR answer for this question names a specific, real technical discovery (not a hypothetical), walks through the concrete diagnostic and remediation steps taken, and ends with a measurable result plus a durable process change.

Structured elaboration, using the STAR framework.

• Situation: be specific about the model and the business context (for example, "I was maintaining a resume-screening model used to shortlist candidates for technical interviews").
• Task: state what alerted you to a possible problem (a routine subgroup-performance review, a stakeholder complaint, a scheduled fairness audit) rather than implying you were simply looking for trouble in the abstract.
• Action: describe the concrete diagnostic steps (computed subgroup-level selection rates and found a meaningful gap; traced the gap to a specific feature or label-quality issue through a leakage-classifier (train a simple classifier to predict the protected attribute from the remaining features; if it can predict the attribute well above chance, those features are acting as a proxy for it) or correlation check; consulted with legal or a fairness review board before deciding on a fix) and the concrete remediation (removed or transformed the offending feature; retrained with a class-weighting or reweighing adjustment; added a monitoring alert on the specific metric that first surfaced the issue).
• Result: state a real, specific, honestly-scoped outcome (the subgroup gap narrowed from X to Y, or the feature was removed and overall accuracy dropped by a small, stated, acceptable amount) rather than an unverifiable superlative like "we completely eliminated bias."

Worked example shape (illustrative skeleton, not a fabricated specific company story). "While auditing a lending model's approval rates, I found the approval rate for one demographic group was noticeably lower than for others. I ran a correlation and leakage-classifier check and found that a ZIP-code-derived feature was acting as a strong proxy for the group in question. I worked with the compliance team to replace that feature with a broader, less granular regional cost-of-living indicator that retained most of the model's predictive signal, re-validated the subgroup gap had narrowed to within the agreed threshold, and added an automated monthly check on that specific metric so a similar drift would be caught immediately rather than at the next scheduled audit."

Trade-offs and pitfalls. (1) Avoid vague framing like "I noticed some bias and fixed it," since an interviewer specifically wants to hear the diagnostic METHOD, not just the conclusion. (2) Never claim a numeric improvement you cannot actually substantiate; if you do not remember the exact percentage, say so and describe the DIRECTION and rough magnitude honestly rather than inventing a precise number. (3) The "what you changed to prevent recurrence" part of the answer is frequently skipped but is usually what distinguishes a strong senior answer from a merely competent one, since it shows systems thinking beyond the single incident.

Direct answer

Class imbalance and fairness interact because the two kinds of "minority" (the minority OUTCOME class and a minority DEMOGRAPHIC group) are often correlated in practice: whichever group has fewer positive-label examples gets a model that is trained on less signal for that exact combination, so it learns that group's decision boundary less reliably and typically pays for it with a higher error rate specifically for that group, not spread evenly across everyone. The standard fixes (oversampling, undersampling, synthetic minority generation as preprocessing, and class-weighted loss as an in-training method) address the raw imbalance in the LABEL, but every one of them needs to be evaluated by GROUP, not only overall, because a fix that improves the minority-class recall in aggregate can still leave (or even worsen) a specific group's error rate if that group is disproportionately part of the class the fix under- or over-corrects.

Structured elaboration

Why imbalance is a fairness issue, not just an accuracy issue. A model trained on an imbalanced outcome (say, 5% positive rate overall) already tends to under-predict the minority class in aggregate, because minimizing average loss rewards defaulting toward the majority class. If the minority OUTCOME class is also disproportionately concentrated in a demographic minority group (a common real-world pattern: a rare positive outcome, like loan default or a safety incident, occurring at different base rates by group, or simply a demographic minority group being a numerical minority in the dataset overall), the model has doubly little signal for exactly the (group, outcome) combination that would let it learn that group well, and the resulting error-rate gap by group looks, from a fairness dashboard, indistinguishable from a "bias" in the traditional sense, even though its proximate cause is a data-volume problem, not a labeling or sampling defect.

Three pre-processing strategies.

1. Random oversampling of the minority class. Duplicate minority-class examples until the training set is closer to balanced. Pros: simple, preserves every original data point, works with any downstream model. Cons for fairness-sensitive applications: duplication does not add new information, it just re-weights existing points, so it can amplify whatever noise or labeling error already exists in those specific minority-class examples; more importantly, if the minority class is concentrated in one demographic group, oversampling that class effectively oversamples that group's specific examples, which can improve that group's recall but can also entrench any measurement or label bias already present in that group's data, since the same flawed examples are simply repeated more often rather than corrected.
2. Random undersampling of the majority class. Drop majority-class examples until the classes are closer to balanced. Pros: also simple, and unlike oversampling it does not risk overfitting on duplicated points. Cons: it discards real information, which is a bigger cost when the discarded majority-class examples are themselves needed to represent a demographic group well (undersampling a class that a MAJORITY group dominates can inadvertently shrink that group's overall representation in training too, if the classes are not balanced independently within each group).
3. Synthetic minority oversampling (SMOTE and its variants). Generate new, synthetic minority-class examples by interpolating between real minority-class examples in feature space, rather than duplicating them exactly. Pros: adds genuinely new (if synthetic) points rather than repeating the same ones, which reduces the overfitting risk of plain oversampling. Cons for fairness: interpolating between minority-class examples ACROSS groups can create synthetic examples that blend feature patterns from different groups in ways that do not correspond to any real person, and if the minority class's within-group feature distributions differ across groups, naive SMOTE applied without group-stratification can generate synthetic points that are unrepresentative for both groups, or can silently smooth over a genuine, real fairness-relevant difference between how the minority outcome presents in each group.

One in-training method: class-weighted (cost-sensitive) loss. Instead of resampling the data, reweight the LOSS FUNCTION so misclassifying a minority-class example costs more than misclassifying a majority-class example (for example, weighting each class inversely proportional to its frequency, or a focal-loss-style down-weighting of easy, well-classified majority examples). Pros: uses every real data point exactly once per epoch (no duplication, no information loss, no synthetic data), integrates cleanly into standard training loops via a single loss-function argument, and is straightforward to extend to a JOINT (class, group) weighting scheme that targets exactly the doubly-underrepresented cells directly, rather than only the class margin. Cons for fairness-sensitive applications: the weighting scheme is a hyperparameter choice (inverse frequency is a common default, not a universal law), and an aggressive class weight can push the model toward over-predicting the minority class broadly, which trades false negatives for false positives in a way that needs to be evaluated by group specifically, since the group that benefits from fewer missed positives is not always the same group that bears the cost of the resulting extra false positives.

Worked example

Consider an employee-attrition prediction model where "will leave within 6 months" is the rare positive class (roughly 8% base rate) and a demographic minority group makes up a smaller share of the workforce, so the (minority-group, will-leave) cell has very few training examples. A team applying the standard toolkit here would: use pre-processing balancing (a joint class-and-group-stratified oversampling or SMOTE, so the correction targets the specific underrepresented (group, outcome) cell rather than the class margin alone), pair it with a fairness-aware training objective (a class-weighted loss with per-group weight terms, so the model is penalized specifically for missing the minority group's rare positive cases, not just the rare class overall), add post-hoc calibration by group (since a rebalanced training distribution changes the model's raw output scale, the deployed model needs group-wise recalibration: for each group, build a reliability diagram (bin the model's predicted probabilities and compare each bin's average prediction against that bin's actual outcome rate) to confirm a calibration gap exists, then refit the mapping from raw score to probability separately per group, using either Platt scaling (a 2-parameter logistic fit on the raw score) or isotonic regression (a flexible monotone step function fit via pool-adjacent-violators), whichever the group's calibration data size supports, so the corrected model's predicted probabilities remain meaningful after the rebalancing), and ship with deployment guardrails (a minimum-sample-size floor before trusting any per-group attrition-risk score, and a human-review step for the smallest groups where even a rebalanced model's estimates carry real uncertainty). Every step in that chain exists specifically because a single technique (say, oversampling alone) would improve the rare-outcome recall in aggregate while leaving the deployed probabilities miscalibrated and the smallest group's estimates unreliable.

Regulatory caution: oversampling a class correlated with a protected group. If the rare positive class is disproportionately concentrated in one demographic group (a realistic pattern: in a hiring-rejection or credit-default model, an outcome correlated with historical circumstances that themselves correlate with a protected characteristic), oversampling that class to fix imbalance can, as a side effect, systematically amplify that group's weight in training specifically around the sensitive prediction boundary. This is not automatically improper, better recall on a genuinely underrepresented group is often the point, but it needs the same disparate-impact scrutiny as any other modeling choice that changes decision rates by group: document why the imbalance correction was applied, measure its effect on the selection-rate ratio and calibration by group before and after, and be prepared to explain to legal or compliance that the correction targeted a genuine data-volume problem rather than an outcome-shopping exercise that happened to move the numbers in a convenient direction.

Trade-offs and pitfalls

• Evaluating any of these four techniques only on overall metrics (aggregate recall, aggregate F1) is the single most common mistake; report the same metric BY GROUP before and after the fix, since a technique can look like an unambiguous win in aggregate while making one group's outcome worse.
• SMOTE-style interpolation without group stratification is an easy default to reach for that can quietly wash out a real, fairness-relevant difference between groups; when the sensitive attribute is known, generate synthetic examples within each group separately, not across the pooled minority class.
• A class-weighted loss is often preferable to resampling specifically because it does not distort the effective sample size the model "sees" (no duplicated or synthetic rows to account for in downstream diagnostics), but it still needs the SAME post-hoc calibration check as any other imbalance fix, since a reweighted loss changes what the model's raw score actually represents.
• Treating a class-imbalance-driven group gap as if it were the SAME problem as a group-level labeling or sampling bias (meaning the ground-truth labels themselves were assigned inconsistently by group, for example because one group's cases were reviewed by different annotators, under different criteria, or with systematically worse instrumentation or follow-up) risks applying the wrong fix; a genuinely data-volume-driven gap is well-served by the techniques here, but a gap caused by a biased LABEL itself needs label-bias remediation: auditing a sample of each group's labels against a common, documented standard, re-adjudicating disagreements (ideally with reviewers blind to the sensitive attribute), and correcting or discarding labels found to be wrong, not resampling or reweighting a label that is systematically incorrect for one group in the first place.

A strong answer opens by naming the three definitions and stating plainly that they generally cannot all hold at once when base rates differ across groups.

Structured elaboration

Loan example. Say a bank approves loans with a risk score.

• Demographic parity means the same fraction of applicants in each demographic group gets approved, even if the groups have different true default rates.
• Equalized odds means that among applicants who would actually repay, the approval rate is the same across groups (equal TPR), and among those who would default, the rejection rate is the same across groups (equal FPR).
• Calibration means that a 0.2 default-risk score means a genuine 20% default probability whether the applicant is in group A or group B.

Worked example. If group A has a true default rate of 10% and group B has a true default rate of 30%, a single calibrated score function will naturally assign more high scores to group B. Forcing demographic parity on top of that calibration would require either denying good group-A applicants or approving bad group-B applicants purely to match rates, which breaks calibration. This is not a hypothetical: it is the mathematical content of the impossibility result once you fix differing base rates.

Trade-offs and pitfalls. (a) A regulator asking for "equal treatment across groups" usually means demographic parity or equalized odds, not calibration, so lean there. (b) A downstream risk-scoring use case (setting an interest rate, sizing a reserve) needs calibration, because a wrongly-calibrated score misprices risk for an entire group even if selection rates look fair. (c) The most common mistake is treating these three as compatible variations on "fairness" rather than as genuinely conflicting design choices; picking one is a policy decision, not a purely technical one, and should be made with legal and business stakeholders, not unilaterally by the model team.

Direct answer

Operationalizing fairness at staff level means designing five interlocking pieces, team structure, KPIs, training/playbooks, legal involvement, and incident response, plus the incentive structure that determines whether product teams actually use any of it. The central-versus-embedded team-structure choice is the highest-leverage decision because it shapes how every other piece gets staffed and enforced, and a hybrid (a small central team owning standards, tooling, and the approval gate, with embedded points-of-contact executing within each product team) usually beats either pure extreme, because pure-central is consistent but slow and easy to bottleneck, while pure-embedded is fast but produces inconsistent practice across teams with no one accountable for the whole picture.

Structured elaboration

Team structure, with explicit trade-offs.

• Central Responsible-AI (RAI) team: owns the fairness-testing and explainability tooling, sets the gate criteria, and reviews high-risk launches directly. Strength: consistency (every team is held to the same bar) and deep, concentrated expertise. Weakness: becomes a bottleneck as the number of product teams grows, and a central team without embedded context can misjudge domain-specific nuance (a fairness concern in lending looks different from one in content ranking).
• Embedded experts: a fairness-literate engineer or scientist sits inside each product team and owns fairness work locally. Strength: fast, context-aware, no queue to wait behind. Weakness: practice drifts between teams (different metrics, different thresholds, different rigor), and an embedded expert can face pressure from their own team's launch incentives in a way a central reviewer, structurally outside that team's roadmap pressure, does not.
• Hybrid: central team owns standards, shared tooling, the gate criteria, and reviews only the highest-risk launches directly; embedded points-of-contact in each product team run the standard checks locally for lower-risk work and escalate to central review only when a metric fails or the use case is novel. This captures most of central's consistency without most of embedded's bottleneck, at the cost of needing clear escalation criteria so "low risk, handle locally" doesn't quietly become "never actually reviewed."

KPIs to track, split by what they actually measure: process KPIs (percentage of launches that went through the gate before shipping, median time-to-approval, count of models with a current, non-stale model card) versus outcome KPIs (count of confirmed fairness incidents per quarter, mean time-to-detect and mean time-to-remediate a confirmed violation, disparate-impact ratio trend across the model portfolio). Process KPIs alone create an incentive to game the process (fast approvals, not necessarily fair outcomes); outcome KPIs alone arrive too late to manage proactively (you only see the incident after it already happened). Track both, and treat a good process KPI trend with a bad outcome KPI trend as a signal the process itself needs revision, not just enforcement.

Training and playbooks. Generic annual fairness training has low retention; the higher-leverage investment is a per-role playbook ("if you are an ML engineer shipping a scoring model, here is exactly which checks apply and how to run them") plus a lightweight, searchable decision log of past fairness reviews, so a team facing a new but similar situation can find precedent instead of re-deriving the reasoning from scratch.

Legal involvement, calibrated to risk tier: legal reviews the GATE CRITERIA themselves (what floor, what documentation is required, for which jurisdictions) as a standing function, but is looped into individual launch reviews only above a defined risk threshold (any use case touching employment, credit, housing, or another legally protected domain), not every launch; involving legal in every low-risk launch review both slows delivery and wastes legal's attention on cases with genuinely low exposure.

Incident response, tied back into the SAME gate and KPI system: a confirmed fairness incident should trigger not just a fix to the specific model but a review of whether the gate criteria or process failed to catch it, feeding back into the process KPIs above.

A concrete organizational mechanism that operationalizes several of the above at once: a cross-functional model-risk committee. Rather than leaving "is this launch high-risk enough for full central review" to informal judgment, a standing committee (RAI lead, legal, a rotating product representative, and a senior engineer from the launching team) meets on a fixed cadence to make the risk-tier and approval-checkpoint calls for any launch the embedded/local review escalates, and separately audits a sample of "locally approved, not escalated" launches to check whether the escalation criteria themselves are working. This gives the hybrid structure its actual teeth: without a committee empowered to say no, "escalate to central review" can quietly become optional under launch-date pressure.

Incentives for product teams. The single biggest determinant of whether any of the above gets used in practice is whether product teams are rewarded or merely tolerated for engaging with it: if a team's ship-date OKRs do not account for gate review time, the gate becomes something to route around under pressure; the fix is making "passed fairness review on schedule" a visible, credited part of the launch process (the same way a security review or a performance benchmark already is in most organizations), not a separate, unfunded obligation layered on top of the real goals.

Worked example

A weighted decision matrix comparing the three structures on five criteria, the kind of comparison a staff engineer would actually bring to a leadership decision rather than a verbal preference:

criteria_weights = {
    "consistency_across_teams": 0.25, "speed_of_product_iteration": 0.20,
    "depth_of_domain_expertise": 0.20, "cost_to_scale": 0.15, "escalation_clarity": 0.20,
}
structures = {
    "central_RAI_team": {"consistency_across_teams": 9, "speed_of_product_iteration": 4,
        "depth_of_domain_expertise": 9, "cost_to_scale": 4, "escalation_clarity": 9},
    "embedded_experts": {"consistency_across_teams": 4, "speed_of_product_iteration": 9,
        "depth_of_domain_expertise": 6, "cost_to_scale": 6, "escalation_clarity": 4},
    "hybrid_central_standards_embedded_execution": {"consistency_across_teams": 8,
        "speed_of_product_iteration": 7, "depth_of_domain_expertise": 7, "cost_to_scale": 6,
        "escalation_clarity": 8},
}
def weighted_score(scores):
    return sum(criteria_weights[k] * v for k, v in scores.items())

scored = sorted(structures.items(), key=lambda kv: weighted_score(kv[1]), reverse=True)
for name, scores in scored:
    print(f"{name:<45} weighted_score={round(weighted_score(scores), 2)}")

Executed output:

hybrid_central_standards_embedded_execution   weighted_score=7.3
central_RAI_team                              weighted_score=7.25
embedded_experts                              weighted_score=5.7

The scores themselves are the staff engineer's own assumed judgment calls for a mid-size, multi-product org, stated as assumptions rather than measured facts, but the ARITHMETIC that turns five separate criteria into a ranked recommendation is exactly reproducible. The margin between hybrid (7.3) and pure-central (7.25) is small enough that the actual deciding factor in a real org would be organization-specific (how many product teams, how mature is central tooling already, how much launch-velocity pressure exists), which is itself a useful output: the matrix shows this is a close call between two of the three options, not an obvious slam dunk, and pure-embedded is clearly weaker on this org's stated weights.

Trade-offs and pitfalls

The most common failure in a hybrid structure is leaving the escalation criteria vague ("escalate anything risky"), which under launch-date pressure resolves to almost nothing getting escalated; the criteria need to be as concrete as the gate thresholds themselves (specific use-case categories, specific metric-breach conditions) and audited periodically, which is exactly the risk committee's second function above. A second pitfall is over-indexing on outcome KPIs (incident count) without accounting for the fact that a genuinely improving process can show a TEMPORARY rise in confirmed incidents simply because better monitoring is catching things that were previously invisible; the KPI trend needs to be read alongside detection-capability changes, not treated as a pure signal of underlying model quality. A third pitfall is under-resourcing the central team relative to the number of product teams it serves in a hybrid model, which quietly degrades the hybrid back into embedded-in-practice (local teams self-approve because central review queues are too slow to use); central staffing needs to scale with the number of escalations actually occurring, not stay fixed at launch headcount. Finally, incentive design is the piece most often skipped entirely in favor of policy and tooling, and it is usually the actual root cause when a well-designed process gets bypassed: if leadership does not visibly protect gate-review time against ship-date pressure, no amount of KPI dashboarding or committee structure will make product teams treat the process as anything other than optional friction.

Direct answer

Local explanations describe a SINGLE prediction (why did this applicant get denied); global explanations describe the model's OVERALL behavior across the population (which features matter most on average). SHAP and LIME are the standard local pair; permutation importance and partial dependence plots (PDP) are the standard global pair. For a regulated financial product, prioritize local explanations wherever an individual decision is contested or requires an adverse-action notice, and global explanations for model validation, ongoing monitoring, and demonstrating to a regulator that the model behaves sensibly in aggregate; a single auditor-facing dashboard should show both, side by side, because a regulator's actual question is usually "is this model sound overall, AND can you explain this specific case," not one or the other.

Structured elaboration

Local methods, two concrete techniques. SHAP (Shapley Additive Explanations) attributes a specific prediction to its input features using cooperative game theory's Shapley value, giving an additive per-feature contribution that sums exactly to the difference between this prediction and the model's average output. LIME (Local Interpretable Model-agnostic Explanations) fits a simple, weighted local surrogate model in a small neighborhood around the instance and reads the surrogate's coefficients as the explanation; unlike SHAP, it makes no game-theoretic optimality claim, but is fully model-agnostic and often faster to compute for a single instance.

Global methods, two concrete techniques. Permutation importance measures how much a model's performance metric degrades when a single feature's values are randomly shuffled, holding everything else fixed; a feature the model actually relies on will show a real performance drop, a feature it ignores will show none. Partial dependence plots show the model's average predicted output as one feature is swept across its range with all other features held at their observed distribution, revealing the SHAPE of a feature's overall relationship with the prediction (linear, threshold, non-monotonic), which a single importance number cannot show.

When to prioritize which, in a regulated financial product. Prioritize LOCAL explanations whenever an individual is affected by a specific decision (an adverse-action notice, a dispute, an internal case review), because the applicable regulation (again, ECOA/Regulation B in a US lending context) requires an explanation of THIS decision, not a statement about the model in general. Prioritize GLOBAL explanations for model validation before deployment, ongoing model-risk monitoring, and demonstrating overall soundness to a regulator or internal model-risk-management function, because these audiences need to know the model behaves reasonably across the whole population, not just that any one case can be explained after the fact.

Presenting both in a single auditor dashboard. A workable layout: a global panel up top (permutation importance ranking, PDP curves for the top few features, a fairness-metric summary broken out by protected group) that answers "does this model make sense overall," and a searchable local panel below (enter an application ID, get its SHAP waterfall and the specific reason codes it would generate) that answers "can you show me this one." Linking the two matters: an auditor should be able to click from a global PDP curve showing an unusual non-monotonic shape straight to individual cases sitting in that unusual region, connecting the aggregate pattern to concrete evidence.

The same local-vs-global distinction, applied specifically to a fairness audit. Global explanations validate a FAIRNESS METRIC (demographic parity, equalized odds) computed across a whole population; local explanations validate a SPECIFIC individual's claim of unfair treatment. A common and costly mistake here is drawing a fairness conclusion from local evidence alone: one applicant's SHAP explanation showing a group-correlated proxy feature ranked high does not establish a population-level disparity, and conversely, a clean population-level fairness metric does not guarantee any one individual's specific explanation is free of a proxy-driven artifact. This is the same failure mode as DEI-adjacent decisions made off a handful of anecdotal cases: local evidence is necessary for individual redress, but global, statistically-powered evidence is what a fairness CLAIM about the model as a whole needs to rest on.

Breadth across explainability techniques and modalities. Beyond the four named above: Integrated Gradients and saliency maps / Grad-CAM attribute predictions for differentiable models (neural networks), the former via a path integral of gradients from a baseline to the input, the latter via gradient-weighted activation maps, most commonly used for image and text deep-learning models. Counterfactual explanations answer "what is the smallest change to this input that would flip the prediction," directly actionable and increasingly favored in regulated contexts precisely because it is phrased as a recourse, not just an attribution. TCAV (Testing with Concept Activation Vectors) tests whether a HUMAN-DEFINED CONCEPT (not a raw input feature) influences a neural network's prediction, useful when the features practitioners care about are not literal input columns. Surrogate models (fitting an interpretable model, like a shallow decision tree, to approximate a complex model's GLOBAL behavior, distinct from LIME's local surrogate) round out a genuinely broad toolkit spanning tabular, image, and text modalities.

The five-technique feature-importance catalog, computed side by side. Mean Decrease in Impurity (MDI, built into tree ensembles, fast but known to inflate importance for high-cardinality or noisy continuous features due to how split-search bias interacts with impurity reduction), permutation importance (model-agnostic, measures actual predictive reliance, more expensive since it requires re-scoring), SHAP (additive, locally-exact, aggregable to a global ranking by averaging), LIME (inherently local; a "global" LIME ranking, as computed below, is an AVERAGE of many local explanations, which is a meaningfully different and noisier construction than a method designed to be global from the start), and standardized regression coefficient magnitude (requires a linear or logistic model, or a linear proxy model, but gives a directly interpretable per-unit-of-standard-deviation effect size).

Worked example

All five feature-importance techniques computed on the SAME model, SAME data, including a pure-noise control feature that every honest method should rank last:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
import shap

np.random.seed(15)
n = 4000
income = np.random.normal(55000, 18000, n)
credit_score = np.random.normal(660, 65, n)
debt_ratio = np.clip(np.random.normal(0.32, 0.15, n), 0, 1)
delinquencies = np.random.poisson(0.8, n)
random_noise = np.random.normal(0, 1, n)  # a control feature with NO true relationship to the label

income_z = (income - income.mean()) / income.std()
credit_z = (credit_score - credit_score.mean()) / credit_score.std()
debt_z = (debt_ratio - debt_ratio.mean()) / debt_ratio.std()
delinq_z = (delinquencies - delinquencies.mean()) / delinquencies.std()
logit = 0.0 + 0.7*income_z + 0.6*credit_z - 0.9*debt_z - 0.5*delinq_z + np.random.normal(0,1,n)
approved = np.random.binomial(1, 1/(1+np.exp(-logit)))
print(f"base approval rate = {approved.mean():.3f}")

feature_names = ["income", "credit_score", "debt_ratio", "delinquencies", "random_noise"]
X = np.column_stack([income, credit_score, debt_ratio, delinquencies, random_noise])
Xtr, Xte, ytr, yte = train_test_split(X, approved, test_size=0.3, random_state=0, stratify=approved)

rf = RandomForestClassifier(n_estimators=300, max_depth=6, random_state=0).fit(Xtr, ytr)

# 1) MDI: mean decrease in impurity, built into the trained random forest
mdi = rf.feature_importances_

# 2) Permutation importance: drop in held-out accuracy when a feature's values are shuffled
perm = permutation_importance(rf, Xte, yte, n_repeats=20, random_state=0, scoring="accuracy")
perm_importance = perm.importances_mean

# 3) SHAP: mean |SHAP| on the held-out set (TreeExplainer)
explainer = shap.TreeExplainer(rf)
sv = explainer.shap_values(Xte)
sv_for_class1 = sv if sv.ndim == 2 else sv[:, :, 1]
mean_abs_shap = np.abs(sv_for_class1).mean(axis=0)

# 4) LIME: average |local coefficient| over a sample of test instances (a common way to roll
# many local explanations up into one global-ish summary; each individual LIME call is local)
def lime_tabular_coef(instance, X_train, predict_fn, n_samples=800, kernel_width=1.0, seed=0):
    from sklearn.linear_model import Ridge
    rng_local = np.random.default_rng(seed)
    feat_std = X_train.std(axis=0)
    noise = rng_local.normal(0, 1, size=(n_samples, len(instance))) * feat_std
    samples = instance + noise
    samples[0] = instance
    preds = predict_fn(samples)
    dist = np.sqrt((((samples - instance) / feat_std) ** 2).sum(axis=1))
    weights = np.exp(-(dist ** 2) / (kernel_width ** 2))
    ridge = Ridge(alpha=1.0).fit(samples, preds, sample_weight=weights)
    return np.abs(ridge.coef_)

sample_idx = np.random.default_rng(0).choice(len(Xte), size=40, replace=False)
lime_coefs = np.array([
    lime_tabular_coef(Xte[i], Xtr, lambda s: rf.predict_proba(s)[:, 1], seed=i)
    for i in sample_idx
])
mean_abs_lime = lime_coefs.mean(axis=0)

# 5) Coefficient magnitude: standardized logistic regression coefficients (comparable scale
# across features because inputs are standardized first)
Xtr_std = (Xtr - Xtr.mean(axis=0)) / Xtr.std(axis=0)
logreg = LogisticRegression().fit(Xtr_std, ytr)
coef_magnitude = np.abs(logreg.coef_[0])

def rank(arr):
    return np.argsort(-arr).argsort() + 1  # 1 = most important

print(f"{'feature':16s} {'MDI':>8s} {'perm':>8s} {'SHAP':>8s} {'LIME':>8s} {'|coef|':>8s}   ranks(MDI/perm/SHAP/LIME/coef)")
for i, name in enumerate(feature_names):
    ranks = tuple(int(v) for v in (rank(mdi)[i], rank(perm_importance)[i], rank(mean_abs_shap)[i], rank(mean_abs_lime)[i], rank(coef_magnitude)[i]))
    print(f"{name:16s} {mdi[i]:8.4f} {perm_importance[i]:8.4f} {mean_abs_shap[i]:8.4f} {mean_abs_lime[i]:8.4f} {coef_magnitude[i]:8.4f}   {ranks}")

print(f"\nrandom_noise final rank across methods (5 = least important, out of 5 features): "
      f"MDI={rank(mdi)[4]}, perm={rank(perm_importance)[4]}, SHAP={rank(mean_abs_shap)[4]}, "
      f"LIME={rank(mean_abs_lime)[4]}, coef={rank(coef_magnitude)[4]}")

Executed output:

base approval rate = 0.507
feature               MDI     perm     SHAP     LIME   |coef|   ranks(MDI/perm/SHAP/LIME/coef)
income             0.2572   0.0382   0.0764   0.0000   0.6107   (2, 3, 3, 5, 2)
credit_score       0.2429   0.0495   0.0838   0.0013   0.5829   (3, 2, 2, 4, 3)
debt_ratio         0.3338   0.0638   0.1083   0.1987   0.7173   (1, 1, 1, 1, 1)
delinquencies      0.0977   0.0146   0.0417   0.0433   0.4766   (4, 4, 4, 2, 4)
random_noise       0.0684  -0.0043   0.0043   0.0107   0.0238   (5, 5, 5, 3, 5)

random_noise final rank across methods (5 = least important, out of 5 features): MDI=5, perm=5, SHAP=5, LIME=3, coef=5

Four of the five methods, MDI, permutation importance, SHAP, and standardized coefficient magnitude, agree closely on the overall ranking (debt_ratio first, income and credit_score close behind, delinquencies fourth) and all four correctly rank the pure-noise control feature dead last. LIME, aggregated by averaging 40 independent local explanations, tells a visibly different and noisier story: it ranks random_noise third, not last, and ranks income, one of the two strongest real predictors by every other method, dead last. This is not a bug in the LIME implementation, it is a direct consequence of what "averaging many local explanations" actually is: each individual LIME call is a noisy, sampling-based local estimate whose coefficients can shift meaningfully from one random perturbation draw to the next, and averaging 40 of them does not converge to the same thing a method DESIGNED to be global (MDI, permutation importance) converges to, especially for a feature like income whose local importance varies a lot depending on where in its range a given instance sits.

Trade-offs and pitfalls

The most consequential mistake in a regulated setting is using a LOCAL method's output to make a GLOBAL fairness claim, or vice versa: a single applicant's SHAP explanation is not evidence about the model's population-level disparity, and a clean population-level demographic-parity number does not certify that any one individual's local explanation is free of a proxy artifact, both directions of this altitude mismatch have shown up in real fairness-audit disputes. A second pitfall, made concrete by the worked example above, is treating an averaged-LIME global ranking as interchangeable with a purpose-built global method like permutation importance or MDI; they can disagree meaningfully, and the disagreement is informative (it tells you LIME's local estimates are unstable in exactly the regions income's importance varies) rather than something to average away. Third, dashboards that show ONLY the global panel or ONLY the local search, not both linked together, force an auditor to context-switch between separate tools to connect an aggregate finding to a specific case, which in practice means the connection often never gets made and questions get answered less rigorously than the underlying data would support.