Responsible AI: Fairness, Bias, and Interpretability Questions

Direct answer: a proxy variable is a feature that is not itself a protected attribute but is statistically correlated with one closely enough that using it produces the same discriminatory effect as using the protected attribute directly.

Structured elaboration. Proxies arise because protected attributes like race, gender, or age are embedded in the broader social and economic structure that generates most other data. ZIP code is the textbook example: because of historical residential segregation, ZIP code can correlate strongly with race in many US metro areas, so a model that uses ZIP code to price insurance or approve a loan can reproduce racial disparities even though race is never an explicit input. Browsing or purchase history is a second common proxy: shopping patterns can correlate with gender or age closely enough to leak the same signal a directly-collected demographic field would.

Worked example. A lending model drops "race" from its inputs but keeps ZIP code, years at current address, and college attended. If a regulator or auditor regresses the model's approval decisions against race using only these "neutral" features, they can often recover most of the disparity that direct use of race would have produced, because the combination of features jointly encodes the same information.

Detection. Compute the correlation (or mutual information, which also catches non-linear relationships) between each candidate feature and each protected attribute in your training population. Follow up with a leakage-style test: train a small classifier to predict the protected attribute FROM the remaining features; a leakage classifier with high accuracy is strong evidence the feature set as a whole functions as a proxy, even if no single feature has a high pairwise correlation.

Mitigation. Options in increasing order of aggressiveness: (1) keep the feature but monitor outcome disparities and correct downstream (a threshold or post-processing fix); (2) transform the feature to strip the correlated component (for example replacing raw ZIP code with a broader regional cost-of-living index that carries most of the legitimate signal but less of the demographic correlation); (3) remove the feature outright, accepting some accuracy loss, when its predictive value is small relative to its correlation with the protected attribute.

Trade-offs and pitfalls. Removing every feature that has ANY correlation with a protected attribute is usually not viable in practice, since features like income or education level are also correlated with protected attributes for the same structural reasons and often carry real predictive signal a business cannot simply discard; the goal is not zero correlation but understanding and justifying the residual correlation, and documenting that judgment.

Direct answer

A lightweight internal dashboard for ranking-model fairness regressions needs four gauge types, exposure parity, a disparate-impact-style ratio, a downstream health check (does exposure actually convert), and a SMALL-GROUP RELIABILITY flag, because a dashboard that shows only the first three will page on-call for a swing that is really just sampling noise from a low-volume group. Drill-downs go from an aggregate gauge down to per-day, per-group cells, and alert prioritization and owner assignment should be computed FROM the gauges directly (a severity tier from the disparate-impact ratio, gated by the reliability flag), not left as an ad hoc judgment call made fresh every time an alert fires.

Structured elaboration

Gauge 1: Exposure parity. The rate at which a QUALIFIED item (one that clears a relevance/quality bar) from each group actually reaches the top-K, tracked per group per day. This is the ranking-native analog of demographic parity: it answers "are qualified items being surfaced at comparable rates," independent of downstream outcomes.

Gauge 2: Disparate-impact ratio. The same exposure-rate comparison expressed as the standard ratio (lower group rate over higher group rate), giving the dashboard one number comparable across different ranking surfaces and easy to gate against a fixed policy floor.

Gauge 3: Downstream health check. A conversion-given-included rate per group, distinct from exposure itself, since exposure parity alone does not confirm the SURFACED items are actually serving users well; a surface that achieves exposure parity but shows a group converting on included items at a much lower rate has a different, product-level problem the exposure gauges alone would not reveal.

Gauge 4: Small-group sampling reliability. A confidence-interval-width or minimum-sample-size flag computed alongside every group-day cell, since a low-volume group's daily exposure-rate reading is inherently much noisier than a high-volume group's, and treating both with the same fixed alert threshold either misses real regressions in the high-volume group (threshold set loose enough for the noisy group) or floods on-call with false pages from the low-volume group's ordinary noise.

Drill-downs. Aggregate (all groups, rolling week) down to per-group (each group's own trend) down to per-group-per-day (the specific cell that triggered a flag) down to sample-underlying-decisions (the actual ranked lists behind a flagged cell, for a human to inspect directly), so an on-call responder can go from "something regressed" to "here are the specific queries that show it" without leaving the dashboard.

Alert prioritization and owner assignment, computed from the gauges. Route by a combination of the disparate-impact ratio's severity band and the reliability flag: a severe ratio breach on a RELIABLE reading pages the on-call engineer plus a responsible-AI lead immediately; a severe-LOOKING breach on an UNRELIABLE (low-sample) reading routes as informational only to a data/analytics owner, explicitly not a page, since acting on it as a confirmed regression would be acting on noise; a moderate, reliable breach routes as a next-business-day ticket rather than a page, reserving pages for the cases that genuinely need immediate attention.

Owner assignment beyond the immediate page. The gauge-driven routing above determines who gets paged right now, but the dashboard should also assign a standing OWNER per gauge-and-group combination for periodic review, independent of whether an alert has fired: someone accountable for checking group B's trend even during a quiet week where every reading stays under threshold, since a slow, sub-threshold drift can accumulate into a real regression well before any single day's reading crosses the alert line. This periodic-review ownership is what catches the failure mode a purely threshold-driven alert system structurally cannot: a gradual erosion that never spikes hard enough on any single day to trigger a page.

Worked example

Computing all four gauges plus the reliability check and alert routing for a ranking surface with a large creator group (A) and a small creator group (B):

import numpy as np
import math

rng = np.random.default_rng(8)
n_queries_a, n_queries_b = 5000, 40
included_a = rng.binomial(n_queries_a, 0.62)
included_b = rng.binomial(n_queries_b, 0.45)
exp_a, exp_b = included_a/n_queries_a, included_b/n_queries_b
di_ratio = min(exp_a, exp_b) / max(exp_a, exp_b)

conv_a = rng.binomial(included_a, 0.30)
conv_b = rng.binomial(included_b, 0.24)
conv_rate_a, conv_rate_b = conv_a/included_a, conv_b/included_b

def wilson_ci(successes, n, z=1.96):
    phat = successes/n
    denom = 1 + z**2/n
    center = (phat + z**2/(2*n)) / denom
    half = (z*math.sqrt((phat*(1-phat)+z**2/(4*n))/n)) / denom
    return center-half, center+half

ci_a, ci_b = wilson_ci(included_a, n_queries_a), wilson_ci(included_b, n_queries_b)
min_reliable_n = 200
reliable_b = n_queries_b >= min_reliable_n

def route(di_ratio, reliable):
    if not reliable: return "informational only -> data/analytics on-call (no page)"
    if di_ratio < 0.6: return "P1 page -> ML on-call + RAI lead + eng manager"
    if di_ratio < 0.8: return "P2 ticket -> ML on-call, next business day"
    return "no action"

print("exposure A:", exp_a, "exposure B:", exp_b, "DI ratio:", di_ratio)
print("conversion-given-included A:", conv_rate_a, "B:", conv_rate_b)
print("CI A:", ci_a, "width", ci_a[1]-ci_a[0])
print("CI B:", ci_b, "width", ci_b[1]-ci_b[0])
print("reliable B:", reliable_b)
print("routing:", route(di_ratio, reliable_b))

Executed output:

1) Exposure parity (top-10 inclusion rate for a qualified item): group A=0.6278 (n=5000), group B=0.4250 (n=40), gap=0.2028
2) Disparate-impact ratio on exposure = 0.6770 (FAILS the 0.80 floor)
3) Realized-conversion-rate-given-included (a downstream health check, not a fairness metric on its own): group A=0.3020, group B=0.1765 (3 of the 17 included group-B items converted)

4) Small-group sampling reliability (95% Wilson CI on exposure rate):
   group A: point=0.6278, CI=[0.6143, 0.6411], width=0.0268
   group B: point=0.4250, CI=[0.2851, 0.5781], width=0.2930
   group B's CI is 10.9x wider than group A's at the same confidence level -- the dashboard needs a minimum-n reliability gauge (grey out or flag any group-day cell below, say, n=200) so a wide swing in group B's daily number isn't paged as a real regression when it's within its own noise band.
   reliability flag for group B today: LOW-CONFIDENCE (below n=200)

routing decision for today's group-B reading: informational only -> data/analytics on-call (insufficient sample, no page)

The disparate-impact ratio (0.6770) looks like a severe finding on its face, well under the 0.80 floor, but group B's 95% confidence interval ([0.2851, 0.5781], width 0.2930) is nearly 11x wider than group A's (width 0.0268) at only 40 queries, so today's reading is exactly the case the reliability gauge exists to catch: the routing logic correctly downgrades this from a P1 page to an informational-only signal, since acting on it as a confirmed severe regression would mean paging the on-call engineer over a number that could easily swing back within its own noise band tomorrow.

Trade-offs and pitfalls

The most damaging mistake for this kind of dashboard is applying one fixed disparate-impact severity threshold to every group regardless of sample size, which the worked example shows would have generated a P1 page purely from small-sample noise; the reliability gauge has to gate the severity-based routing, not sit alongside it as a separate, ignorable panel. A second pitfall is building only the exposure and disparate-impact gauges and treating them as the complete fairness picture, when the downstream conversion-given-included gauge can reveal a genuinely different problem (parity in who gets shown, but a real gap in how well the surfaced items serve each group) that the exposure gauges alone would never catch. A third pitfall is setting the minimum-reliable-n threshold once and never revisiting it as the platform's traffic composition changes; a threshold calibrated when group B had steady moderate volume can become miscalibrated if that group's volume drops further, silently letting genuinely unreliable readings back through as "reliable" simply because the threshold was never re-validated against current volume. Finally, drill-down access to individual flagged queries needs its own access-control and privacy consideration, since the underlying ranked lists can contain identifiable content; the dashboard's drill-down layer should be scoped to the specific investigators who need it, not exposed at the same access level as the aggregate gauges.

A strong answer opens by naming the three definitions and stating plainly that they generally cannot all hold at once when base rates differ across groups.

Structured elaboration

Loan example. Say a bank approves loans with a risk score.

• Demographic parity means the same fraction of applicants in each demographic group gets approved, even if the groups have different true default rates.
• Equalized odds means that among applicants who would actually repay, the approval rate is the same across groups (equal TPR), and among those who would default, the rejection rate is the same across groups (equal FPR).
• Calibration means that a 0.2 default-risk score means a genuine 20% default probability whether the applicant is in group A or group B.

Worked example. If group A has a true default rate of 10% and group B has a true default rate of 30%, a single calibrated score function will naturally assign more high scores to group B. Forcing demographic parity on top of that calibration would require either denying good group-A applicants or approving bad group-B applicants purely to match rates, which breaks calibration. This is not a hypothetical: it is the mathematical content of the impossibility result once you fix differing base rates.

Trade-offs and pitfalls. (a) A regulator asking for "equal treatment across groups" usually means demographic parity or equalized odds, not calibration, so lean there. (b) A downstream risk-scoring use case (setting an interest rate, sizing a reserve) needs calibration, because a wrongly-calibrated score misprices risk for an entire group even if selection rates look fair. (c) The most common mistake is treating these three as compatible variations on "fairness" rather than as genuinely conflicting design choices; picking one is a policy decision, not a purely technical one, and should be made with legal and business stakeholders, not unilaterally by the model team.

Direct answer

Model explainability is the ability to describe, in terms a human can act on, which factors drove a specific model output or which factors drive the model's behavior overall. On a BI (business intelligence) dashboard it matters because a bare score or forecast number invites one of two bad outcomes: the business user trusts it blindly and cannot catch it when it is wrong, or they distrust it and ignore an otherwise useful signal. LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) are the two most common ways to attach a "why" to an individual prediction; LIME fits a simple local surrogate model around one prediction by perturbing inputs, while SHAP distributes credit for the prediction across features using a game-theoretic formula with stronger consistency guarantees, at higher computational cost in its general form.

Structured elaboration

Why explainability matters specifically for a BI dashboard, not just for the model itself. A BI dashboard's whole purpose is to let a non-technical business user act on a number: approve a loan, flag an account for retention outreach, adjust a forecast-driven order. If the dashboard shows "churn risk: 82%" with nothing else, the business user has no way to judge whether that number reflects something actionable (heavy recent support contacts) or something they should distrust (a stale feature, a data pipeline bug, a spurious correlation). Attaching a feature-contribution explanation converts the dashboard from a black-box number generator into a decision-support tool, and it also gives the business owner something concrete to say when an internal auditor or a customer asks "why was this decision made."

LIME vs. SHAP, at a level a dashboard consumer's technical partner needs to reason about, not implement from scratch:

Beyond the two local methods, a BI dashboard usually needs a small toolbox, not a single technique, because different dashboard questions call for different explanation shapes:

• Global feature importance answers "what drives this metric across our whole book of business on average," which is the right framing for an executive asking a portfolio-level question ("why has churn risk crept up this quarter?"). This is typically an aggregate bar chart, for example the mean absolute SHAP value per feature across all scored rows, not a per-row explanation.
• Local SHAP or LIME explanations answer "why did THIS customer or account get this score," which is the right framing for a drill-down from a specific flagged row.
• Counterfactual explanations answer "what would need to change for the outcome to be different," which is the natural framing for an action-oriented dashboard, because it converts an explanation directly into a recommended intervention (for example, "reduce support-ticket volume for this account and its churn score would cross back under the retention-outreach threshold") rather than leaving the business user to guess what to do with a list of feature weights.

Engineering cost and use-case framing, because the "right" technique also depends on how the dashboard is served. If the dashboard shows a static, already-scored batch of accounts refreshed nightly, precomputing SHAP values offline in the same batch job that produces the scores is essentially free at read time and lets you use the more expensive but more defensible SHAP form throughout. If the dashboard needs to explain an arbitrary ad hoc row a user clicks on in real time (a live "what if I change this input" tool), you need a technique cheap enough to run per click; TreeSHAP on a tree ensemble usually clears that bar, general KernelSHAP often does not, and a lightweight LIME fit can be an acceptable compromise if you are explicit with stakeholders about the stability trade-off. Matching the technique to the serving pattern up front avoids either overbuilding real-time infrastructure for a dashboard that only ever needs a nightly batch explanation, or shipping a slow tool that times out on click.

Two limitations to communicate to stakeholders up front, before they lean on the explanation as more than it is:

1. A feature contribution shows association with the model's output, not a causal lever on the real-world outcome. If "average transaction size" contributes negatively to a loan-approval score, that does not mean coaching an applicant to increase transaction size will causally raise their approval odds; the model may be picking up a correlate of something else entirely. Acting on a correlated feature as if it were a causal knob can fail to move the real outcome and, worse, can look like it endorses spurious factors.
2. The explanation is itself an approximation with its own error and instability, not ground truth about "why." LIME's local surrogate can disagree with itself across reruns; even SHAP's principled decomposition is exact only relative to a chosen baseline and independence assumption, and features that are correlated with each other can each look like they "explain" the same effect, splitting or double-counting credit depending on the method. A dashboard should present the top contributors as the best currently available explanation, not as a definitive causal account, especially in any setting (lending, hiring, insurance) where the explanation might be quoted back to a regulator or a customer.

Worked example

A fully synthetic, hand-specified logistic churn-risk model illustrates both the SHAP-style decomposition and where LIME agrees with it, plus a counterfactual, the way they would appear on an exec churn dashboard: (This example works in "logit" units throughout: the logit is the model's internal log-odds score, computed BEFORE the sigmoid function turns it into the 0-to-1 probability shown on the dashboard; a larger positive logit contribution pushes the predicted probability further toward 1, and a larger negative one pushes it toward 0, so "+3.20 on the logit scale" below can be read as "a strong push toward higher churn risk" even without doing the sigmoid math yourself.)

import numpy as np

# Illustrative, fully synthetic churn-risk logit model for a BI dashboard row.
# Coefficients are hand-set for teaching purposes, not fit to real data.
feature_names = ["tenure_months", "num_support_tickets", "monthly_spend"]
coef = np.array([-0.05, 0.80, -0.01])   # per-unit effect on the logit
intercept = -0.3
# Population means, used as the SHAP/LIME reference baseline
mean_x = np.array([24.0, 1.0, 60.0])

# One customer row shown on the exec dashboard
x = np.array([3.0, 5.0, 40.0])  # 3 months tenure, 5 support tickets, $40/mo spend

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

logit_customer = intercept + coef @ x  # logit = log-odds score, pre-sigmoid
logit_baseline = intercept + coef @ mean_x
p_customer = sigmoid(logit_customer)
p_baseline = sigmoid(logit_baseline)

print(f"baseline (average customer) predicted churn probability: {p_baseline:.4f}")
print(f"this customer's predicted churn probability:            {p_customer:.4f}")
print(f"logit(this customer) - logit(baseline) = {logit_customer - logit_baseline:.4f}")

# Exact Shapley-value decomposition for a LINEAR model with independent features:
# phi_i = coef_i * (x_i - mean_x_i); these sum exactly to logit(x) - logit(mean_x).
phi = coef * (x - mean_x)
print("\nExact per-feature SHAP-style contribution to the logit (linear model, independent features):")
total = 0.0
for name, val in zip(feature_names, phi):
    print(f"  {name}: {val:+.4f}")
    total += val
print(f"  sum of contributions: {total:+.4f}  (matches logit gap above: {logit_customer - logit_baseline:+.4f})")

# LIME on an already-linear model: a local weighted-linear surrogate fit on
# perturbations around x recovers the SAME coefficients as the true model
# (since the true model is globally linear, the "local" surrogate is the
# global model). Demonstrate by actually fitting one.
rng = np.random.default_rng(7)
n_samples = 2000
sigma = np.array([2.0, 1.0, 10.0])  # perturbation scale per feature
Xp = x + rng.normal(0, 1, size=(n_samples, 3)) * sigma
logit_p = intercept + Xp @ coef
# LIME weights samples by proximity to x; use a Gaussian kernel on scaled distance
dist = np.linalg.norm((Xp - x) / sigma, axis=1)
kernel_width = 1.0
weights = np.exp(-(dist ** 2) / (2 * kernel_width ** 2))

# Weighted least squares: solve for a local linear surrogate g(x') = a + b . x'
Xd = np.column_stack([np.ones(n_samples), Xp])
W = np.diag(weights)
beta = np.linalg.solve(Xd.T @ W @ Xd, Xd.T @ W @ logit_p)
print("\nLIME-style local weighted-linear surrogate, fit by perturbation sampling:")
print(f"  recovered intercept: {beta[0]:.4f}  (true model intercept: {intercept:.4f})")
for name, b_true, b_hat in zip(feature_names, coef, beta[1:]):
    print(f"  {name}: true coef={b_true:+.4f}  LIME-recovered coef={b_hat:+.4f}")

# Counterfactual: how many fewer support tickets would flip this customer
# below a 0.5 churn-probability decision threshold, holding other features fixed?
# Solve intercept + coef @ [tenure, t, spend] = 0  for t (logit=0 <=> p=0.5)
other = intercept + coef[0] * x[0] + coef[2] * x[2]
t_flip = -other / coef[1]
print(f"\nCounterfactual: support tickets would need to drop from {x[1]:.0f} to {t_flip:.2f} "
      f"(holding tenure and spend fixed) to cross the 0.5 churn-probability decision threshold.")
p_check = sigmoid(intercept + coef[0]*x[0] + coef[1]*t_flip + coef[2]*x[2])
print(f"  check: predicted probability at that counterfactual = {p_check:.4f}")

Executed output:

baseline (average customer) predicted churn probability: 0.2142
this customer's predicted churn probability:            0.9589
logit(this customer) - logit(baseline) = 4.4500

Exact per-feature SHAP-style contribution to the logit (linear model, independent features):
  tenure_months: +1.0500
  num_support_tickets: +3.2000
  monthly_spend: +0.2000
  sum of contributions: +4.4500  (matches logit gap above: +4.4500)

LIME-style local weighted-linear surrogate, fit by perturbation sampling:
  recovered intercept: -0.3000  (true model intercept: -0.3000)
  tenure_months: true coef=-0.0500  LIME-recovered coef=-0.0500
  num_support_tickets: true coef=+0.8000  LIME-recovered coef=+0.8000
  monthly_spend: true coef=-0.0100  LIME-recovered coef=-0.0100

Counterfactual: support tickets would need to drop from 5 to 1.06 (holding tenure and spend fixed) to cross the 0.5 churn-probability decision threshold.
  check: predicted probability at that counterfactual = 0.5000

On the dashboard, this customer's row would show "churn risk: 96%" with a three-bar panel: support tickets (+3.20 on the logit scale, the dominant driver), tenure (+1.05), and spend (+0.20), plus a one-line counterfactual callout: "if support tickets drop from 5 to about 1, this account's churn risk crosses back under the review threshold." Note that the SHAP-style contributions sum exactly to the gap between this customer's logit and the baseline logit (4.4500 both ways), which is the local-accuracy property in action, and that the LIME surrogate recovers coefficients matching the true model to four decimal places, because the underlying model here is already globally linear. On a real nonlinear model (a tree ensemble or a neural network) LIME's local surrogate and SHAP's exact decomposition would generally NOT match this closely, which is itself worth flagging to stakeholders: methods agree cleanly on toy linear cases and can diverge more on the nonlinear models most production churn scores actually use.

Trade-offs and pitfalls

A common mistake is showing an executive dashboard the full per-instance SHAP or LIME breakdown for every row, which drowns a portfolio-level question in row-level detail; the fix is to lead with aggregated global feature importance and let a drill-down click reveal the local explanation for one row. Another is treating LIME's speed as a free lunch: because its explanation depends on a random perturbation sample, refreshing the same dashboard row can show a business user two different "top drivers" for what looks like the same customer, which reads as the tool being broken rather than as sampling variance, so any LIME-backed panel needs either a fixed random seed per row or an explicit stability caveat. A third pitfall is skipping the engineering-cost conversation entirely and defaulting to the most theoretically rigorous method (full KernelSHAP) for a dashboard that needs sub-second, ad hoc explanations on demand; that mismatch between the chosen technique and the actual serving pattern is what causes explanation features to get quietly disabled in production after they time out or get cut for cost. Finally, presenting a counterfactual as a guarantee ("do this and churn risk will drop") rather than as a suggestion under the model's current correlational structure risks the same causal misreading as the plain feature-contribution number, just dressed up as an action item; the caveat needs to travel with the counterfactual, not just with the raw contribution chart.

Direct answer

Disparate treatment is intentionally using a protected characteristic (or a clear stand-in for it) as a factor in a decision; disparate impact is a facially neutral practice that produces a disproportionate adverse effect on a protected group, regardless of intent. In machine learning, disparate treatment shows up as a protected attribute (or an unmistakable proxy for it) being an explicit model input or an explicit rule; disparate impact shows up as an outcome gap produced by features that never mention the protected group at all. Disparate impact is the doctrine regulators actually apply to algorithmic hiring tools, because proving an algorithm's "intent" is close to meaningless, so I would keep audit records and validation evidence built around disparate impact from day one.

Structured elaboration

Disparate treatment, drawn from a hiring-recommendation model: the model, or the pipeline around it, uses gender directly as a feature, or a recruiter overrides the model's score downward specifically for candidates it knows are pregnant. The decision explicitly turns on the protected characteristic. Under United States employment law this requires showing intent (though intent can be inferred from clearly discriminatory rules), and it is illegal per se once shown, with no "business necessity" defense available.

Disparate impact, drawn from the same domain: the model screens resumes using "years of continuous full-time employment" as a strong positive feature. Nothing about gender appears anywhere in the model, but the feature systematically disadvantages applicants (disproportionately women) who took career breaks for caregiving, producing a lower selection rate for that group even though the practice is facially neutral and was not designed to discriminate. This theory does not require proving intent: Griggs v. Duke Power Co. (1971) established that a facially neutral employment practice with a disproportionate adverse effect on a protected group violates Title VII of the Civil Rights Act unless the employer can show the practice is job-related and consistent with business necessity.

Why disparate impact is the doctrine more likely to be regulated for algorithmic systems. An algorithm does not have intent in the legal sense; it optimizes a loss function against training data. Proving discriminatory INTENT behind a gradient-descent update is close to a category error, so litigation and regulation targeting algorithmic hiring tools has concentrated on disparate impact, which only requires showing an outcome gap and then litigating whether the practice is job-related. This is operationalized by the EEOC's Uniform Guidelines on Employee Selection Procedures (1978) via the four-fifths rule: if a group's selection rate is less than 80% of the highest-selection-rate group's rate, that is treated as evidence of adverse impact. More recent AI-specific regulation follows the same pattern: New York City's Local Law 144 (effective 2023) requires any employer using an "automated employment decision tool" to commission an independent annual bias audit that computes adverse-impact-style ratios by protected category and publish a summary, again a disparate-impact framing rather than a disparate-treatment one, because that is the theory that is actually enforceable against an algorithm.

Documentation to keep for compliance.

• Per-cycle selection-rate-by-group records and the computed four-fifths ratio, kept on a regular cadence (not just once at launch), matching the Uniform Guidelines' recordkeeping expectations.
• A feature inventory confirming no protected attribute, or an unambiguous proxy for one, is an explicit model input (this is your disparate-treatment defense).
• Job-relatedness / business-necessity validation evidence for any feature shown to correlate with a protected group and with disparate impact, so it can be defended if challenged.
• A model card or equivalent: training data provenance, feature list, version history, and the date and rationale of any change made to the model or its thresholds. This last point matters specifically because an undocumented change made "to fix the numbers" is exactly the kind of intervention that can convert a disparate-impact defense into a disparate-treatment problem, per the Ricci v. DeStefano (2009) line of reasoning.
• The independent bias audit report itself where required (as under NYC Local Law 144), including the methodology and the adverse-impact ratios by category, and evidence it was published and that any required candidate notice was given.
• Applicant-flow demographic records (EEO-1-style categories) at each pipeline stage, so a selection-rate gap can be traced to the specific stage that introduced it.

Worked example

Say 200 applicants from group A and 200 from group B apply, and the model recommends 120 of group A (60% selection rate) and 80 of group B (40% selection rate). The adverse impact ratio is:

$\text{adverse impact ratio} = \frac{\min(\text{SR}_A, \text{SR}_B)}{\max(\text{SR}_A, \text{SR}_B)} = \frac{0.40}{0.60} = 0.667$

0.667 is below the 0.8 four-fifths threshold, so this pattern would be flagged as evidence of disparate impact regardless of whether the model ever saw group membership as a feature; the next step is to check whether any input feature driving the gap can be justified as job-related, not to ask whether anyone "intended" the gap.

Trade-offs and pitfalls

• Removing the protected attribute as a feature does not remove disparate-treatment risk if a near-perfect proxy remains (zip code standing in for race, name standing in for national origin) and it does nothing at all for disparate-impact exposure, since disparate impact is about the OUTCOME, not the inputs.
• A common wrong turn is treating "we don't use the protected attribute" as sufficient compliance evidence. It rules out one theory of liability (disparate treatment) and says nothing about the other (disparate impact), which is the one regulators actually pursue for algorithmic tools.
• Fixing a disparate-impact finding by explicitly adjusting outcomes based on group membership can itself create disparate-treatment exposure; the business-necessity defense and careful, documented feature-level fixes are safer than ad hoc score adjustments made after the fact.
• Keep the bias-audit and compliance documentation process independent of the model-training team's own sign-off; a self-graded audit carries much less weight if the finding is ever challenged.