Responsible AI: Fairness, Bias, and Interpretability Questions

Direct answer: a proxy variable is a feature that is not itself a protected attribute but is statistically correlated with one closely enough that using it produces the same discriminatory effect as using the protected attribute directly.

Structured elaboration. Proxies arise because protected attributes like race, gender, or age are embedded in the broader social and economic structure that generates most other data. ZIP code is the textbook example: because of historical residential segregation, ZIP code can correlate strongly with race in many US metro areas, so a model that uses ZIP code to price insurance or approve a loan can reproduce racial disparities even though race is never an explicit input. Browsing or purchase history is a second common proxy: shopping patterns can correlate with gender or age closely enough to leak the same signal a directly-collected demographic field would.

Worked example. A lending model drops "race" from its inputs but keeps ZIP code, years at current address, and college attended. If a regulator or auditor regresses the model's approval decisions against race using only these "neutral" features, they can often recover most of the disparity that direct use of race would have produced, because the combination of features jointly encodes the same information.

Detection. Compute the correlation (or mutual information, which also catches non-linear relationships) between each candidate feature and each protected attribute in your training population. Follow up with a leakage-style test: train a small classifier to predict the protected attribute FROM the remaining features; a leakage classifier with high accuracy is strong evidence the feature set as a whole functions as a proxy, even if no single feature has a high pairwise correlation.

Mitigation. Options in increasing order of aggressiveness: (1) keep the feature but monitor outcome disparities and correct downstream (a threshold or post-processing fix); (2) transform the feature to strip the correlated component (for example replacing raw ZIP code with a broader regional cost-of-living index that carries most of the legitimate signal but less of the demographic correlation); (3) remove the feature outright, accepting some accuracy loss, when its predictive value is small relative to its correlation with the protected attribute.

Trade-offs and pitfalls. Removing every feature that has ANY correlation with a protected attribute is usually not viable in practice, since features like income or education level are also correlated with protected attributes for the same structural reasons and often carry real predictive signal a business cannot simply discard; the goal is not zero correlation but understanding and justifying the residual correlation, and documenting that judgment.

A strong answer opens by naming the three definitions and stating plainly that they generally cannot all hold at once when base rates differ across groups.

Structured elaboration

Loan example. Say a bank approves loans with a risk score.

• Demographic parity means the same fraction of applicants in each demographic group gets approved, even if the groups have different true default rates.
• Equalized odds means that among applicants who would actually repay, the approval rate is the same across groups (equal TPR), and among those who would default, the rejection rate is the same across groups (equal FPR).
• Calibration means that a 0.2 default-risk score means a genuine 20% default probability whether the applicant is in group A or group B.

Worked example. If group A has a true default rate of 10% and group B has a true default rate of 30%, a single calibrated score function will naturally assign more high scores to group B. Forcing demographic parity on top of that calibration would require either denying good group-A applicants or approving bad group-B applicants purely to match rates, which breaks calibration. This is not a hypothetical: it is the mathematical content of the impossibility result once you fix differing base rates.

Trade-offs and pitfalls. (a) A regulator asking for "equal treatment across groups" usually means demographic parity or equalized odds, not calibration, so lean there. (b) A downstream risk-scoring use case (setting an interest rate, sizing a reserve) needs calibration, because a wrongly-calibrated score misprices risk for an entire group even if selection rates look fair. (c) The most common mistake is treating these three as compatible variations on "fairness" rather than as genuinely conflicting design choices; picking one is a policy decision, not a purely technical one, and should be made with legal and business stakeholders, not unilaterally by the model team.

Direct answer

A lightweight internal dashboard for ranking-model fairness regressions needs four gauge types, exposure parity, a disparate-impact-style ratio, a downstream health check (does exposure actually convert), and a SMALL-GROUP RELIABILITY flag, because a dashboard that shows only the first three will page on-call for a swing that is really just sampling noise from a low-volume group. Drill-downs go from an aggregate gauge down to per-day, per-group cells, and alert prioritization and owner assignment should be computed FROM the gauges directly (a severity tier from the disparate-impact ratio, gated by the reliability flag), not left as an ad hoc judgment call made fresh every time an alert fires.

Structured elaboration

Gauge 1: Exposure parity. The rate at which a QUALIFIED item (one that clears a relevance/quality bar) from each group actually reaches the top-K, tracked per group per day. This is the ranking-native analog of demographic parity: it answers "are qualified items being surfaced at comparable rates," independent of downstream outcomes.

Gauge 2: Disparate-impact ratio. The same exposure-rate comparison expressed as the standard ratio (lower group rate over higher group rate), giving the dashboard one number comparable across different ranking surfaces and easy to gate against a fixed policy floor.

Gauge 3: Downstream health check. A conversion-given-included rate per group, distinct from exposure itself, since exposure parity alone does not confirm the SURFACED items are actually serving users well; a surface that achieves exposure parity but shows a group converting on included items at a much lower rate has a different, product-level problem the exposure gauges alone would not reveal.

Gauge 4: Small-group sampling reliability. A confidence-interval-width or minimum-sample-size flag computed alongside every group-day cell, since a low-volume group's daily exposure-rate reading is inherently much noisier than a high-volume group's, and treating both with the same fixed alert threshold either misses real regressions in the high-volume group (threshold set loose enough for the noisy group) or floods on-call with false pages from the low-volume group's ordinary noise.

Drill-downs. Aggregate (all groups, rolling week) down to per-group (each group's own trend) down to per-group-per-day (the specific cell that triggered a flag) down to sample-underlying-decisions (the actual ranked lists behind a flagged cell, for a human to inspect directly), so an on-call responder can go from "something regressed" to "here are the specific queries that show it" without leaving the dashboard.

Alert prioritization and owner assignment, computed from the gauges. Route by a combination of the disparate-impact ratio's severity band and the reliability flag: a severe ratio breach on a RELIABLE reading pages the on-call engineer plus a responsible-AI lead immediately; a severe-LOOKING breach on an UNRELIABLE (low-sample) reading routes as informational only to a data/analytics owner, explicitly not a page, since acting on it as a confirmed regression would be acting on noise; a moderate, reliable breach routes as a next-business-day ticket rather than a page, reserving pages for the cases that genuinely need immediate attention.

Owner assignment beyond the immediate page. The gauge-driven routing above determines who gets paged right now, but the dashboard should also assign a standing OWNER per gauge-and-group combination for periodic review, independent of whether an alert has fired: someone accountable for checking group B's trend even during a quiet week where every reading stays under threshold, since a slow, sub-threshold drift can accumulate into a real regression well before any single day's reading crosses the alert line. This periodic-review ownership is what catches the failure mode a purely threshold-driven alert system structurally cannot: a gradual erosion that never spikes hard enough on any single day to trigger a page.

Worked example

Computing all four gauges plus the reliability check and alert routing for a ranking surface with a large creator group (A) and a small creator group (B):

import numpy as np
import math

rng = np.random.default_rng(8)
n_queries_a, n_queries_b = 5000, 40
included_a = rng.binomial(n_queries_a, 0.62)
included_b = rng.binomial(n_queries_b, 0.45)
exp_a, exp_b = included_a/n_queries_a, included_b/n_queries_b
di_ratio = min(exp_a, exp_b) / max(exp_a, exp_b)

conv_a = rng.binomial(included_a, 0.30)
conv_b = rng.binomial(included_b, 0.24)
conv_rate_a, conv_rate_b = conv_a/included_a, conv_b/included_b

def wilson_ci(successes, n, z=1.96):
    phat = successes/n
    denom = 1 + z**2/n
    center = (phat + z**2/(2*n)) / denom
    half = (z*math.sqrt((phat*(1-phat)+z**2/(4*n))/n)) / denom
    return center-half, center+half

ci_a, ci_b = wilson_ci(included_a, n_queries_a), wilson_ci(included_b, n_queries_b)
min_reliable_n = 200
reliable_b = n_queries_b >= min_reliable_n

def route(di_ratio, reliable):
    if not reliable: return "informational only -> data/analytics on-call (no page)"
    if di_ratio < 0.6: return "P1 page -> ML on-call + RAI lead + eng manager"
    if di_ratio < 0.8: return "P2 ticket -> ML on-call, next business day"
    return "no action"

print("exposure A:", exp_a, "exposure B:", exp_b, "DI ratio:", di_ratio)
print("conversion-given-included A:", conv_rate_a, "B:", conv_rate_b)
print("CI A:", ci_a, "width", ci_a[1]-ci_a[0])
print("CI B:", ci_b, "width", ci_b[1]-ci_b[0])
print("reliable B:", reliable_b)
print("routing:", route(di_ratio, reliable_b))

Executed output:

1) Exposure parity (top-10 inclusion rate for a qualified item): group A=0.6278 (n=5000), group B=0.4250 (n=40), gap=0.2028
2) Disparate-impact ratio on exposure = 0.6770 (FAILS the 0.80 floor)
3) Realized-conversion-rate-given-included (a downstream health check, not a fairness metric on its own): group A=0.3020, group B=0.1765 (3 of the 17 included group-B items converted)

4) Small-group sampling reliability (95% Wilson CI on exposure rate):
   group A: point=0.6278, CI=[0.6143, 0.6411], width=0.0268
   group B: point=0.4250, CI=[0.2851, 0.5781], width=0.2930
   group B's CI is 10.9x wider than group A's at the same confidence level -- the dashboard needs a minimum-n reliability gauge (grey out or flag any group-day cell below, say, n=200) so a wide swing in group B's daily number isn't paged as a real regression when it's within its own noise band.
   reliability flag for group B today: LOW-CONFIDENCE (below n=200)

routing decision for today's group-B reading: informational only -> data/analytics on-call (insufficient sample, no page)

The disparate-impact ratio (0.6770) looks like a severe finding on its face, well under the 0.80 floor, but group B's 95% confidence interval ([0.2851, 0.5781], width 0.2930) is nearly 11x wider than group A's (width 0.0268) at only 40 queries, so today's reading is exactly the case the reliability gauge exists to catch: the routing logic correctly downgrades this from a P1 page to an informational-only signal, since acting on it as a confirmed severe regression would mean paging the on-call engineer over a number that could easily swing back within its own noise band tomorrow.

Trade-offs and pitfalls

The most damaging mistake for this kind of dashboard is applying one fixed disparate-impact severity threshold to every group regardless of sample size, which the worked example shows would have generated a P1 page purely from small-sample noise; the reliability gauge has to gate the severity-based routing, not sit alongside it as a separate, ignorable panel. A second pitfall is building only the exposure and disparate-impact gauges and treating them as the complete fairness picture, when the downstream conversion-given-included gauge can reveal a genuinely different problem (parity in who gets shown, but a real gap in how well the surfaced items serve each group) that the exposure gauges alone would never catch. A third pitfall is setting the minimum-reliable-n threshold once and never revisiting it as the platform's traffic composition changes; a threshold calibrated when group B had steady moderate volume can become miscalibrated if that group's volume drops further, silently letting genuinely unreliable readings back through as "reliable" simply because the threshold was never re-validated against current volume. Finally, drill-down access to individual flagged queries needs its own access-control and privacy consideration, since the underlying ranked lists can contain identifiable content; the dashboard's drill-down layer should be scoped to the specific investigators who need it, not exposed at the same access level as the aggregate gauges.

Direct answer

Compute each group's approval rate, divide it by the highest approval rate observed among the groups being compared for that same protected attribute (the "80 percent rule" convention from the United States Equal Employment Opportunity Commission's Uniform Guidelines, which compares every group to whichever group fared best rather than to a hardcoded assumed-privileged group), and flag any ratio below 0.8. Two assumptions have to be stated explicitly before the numbers mean anything: rows with a NULL protected attribute must be excluded from that attribute's denominator rather than silently coerced into a group, and any group below a minimum sample size must be flagged as statistically unreliable rather than given a disparate impact ratio at all, since a tiny group's approval rate is too noisy to support a fairness conclusion either way.

Structured elaboration

Choosing the reference group. The 80 percent rule, as codified in the EEOC Uniform Guidelines on Employee Selection Procedures, defines the disparate impact ratio as a group's selection rate divided by the selection rate of the group with the HIGHEST selection rate among those being compared, not divided by an assumed majority or historically privileged group. This matters because hardcoding "male" or "white" as the denominator bakes in an assumption about which group is doing best that the data itself should determine, and it also means the reference group can legitimately be different for gender than for race, since they are separate comparisons over separate partitions of the same population.

NULL handling. A row with a NULL gender or NULL race cannot be assigned to any group, and it should not be silently dropped from the whole analysis, only from the specific cut (gender or race) where the value is missing; a row with a known race but missing gender should still count toward the race-based disparate impact calculation. Beyond just exclusion, the RATE of missingness itself is worth reporting separately: if protected-attribute data is missing disproportionately for one outcome or one time period, that is itself a data-quality finding that can distort every downstream ratio, and it should be surfaced, not silently absorbed into "excluded rows."

Minimum sample size. A group with a handful of applicants can show an extreme approval rate purely from noise: three approvals out of five applicants is a 60% rate that could easily have been 20% or 100% with one different outcome. Flagging a tiny group as a genuine disparate impact finding invites both false alarms (chasing noise) and false reassurance (a real disparity in a small group failing to clear an alarm threshold purely because there is not enough data to be confident either way). The practical fix is a minimum sample size threshold (commonly 30, informally tied to when a binomial proportion's normal approximation becomes reasonable, though a real analysis should also check a proper confidence interval such as Wilson or Clopper-Pearson rather than relying on the threshold alone) below which the query reports the group as having an insufficient sample rather than a computed ratio, so a "no flag" on a tiny group is never mistaken for a clean bill of health.

Query structure. Compute per-group counts and approval rates for gender and race as two separate partitions (a UNION ALL of two grouped aggregates, tagged by which attribute they belong to), take the maximum approval rate within each attribute's partition as that attribute's reference rate using a window function, then compute the ratio and the flag in one final pass.

Worked example

import sqlite3
import numpy as np

rng = np.random.default_rng(21)
conn = sqlite3.connect(":memory:")
cur = conn.cursor()
cur.execute('''
CREATE TABLE applications (
    id INTEGER PRIMARY KEY,
    outcome INTEGER,       -- 1 = approved, 0 = denied
    gender TEXT,
    race TEXT,
    score REAL,
    applied_at TEXT
)
''')

# Synthetic applicant pool with a deliberate, known disparity:
# - gender: 'male' approval rate ~0.55, 'female' ~0.40 (a real, measurable gap),
#   plus a small 'nonbinary' group (n=18, below a minimum-sample threshold).
# - race: 'white' ~0.55, 'black' ~0.38, 'asian' ~0.56, plus a tiny 'native_american'
#   group (n=12, also below the minimum-sample threshold).
# - a handful of rows have NULL gender/race, to exercise the null-handling rule.
rows = []
rid = 1

def add_group(n, gender, race, approval_rate):
    global rid
    for _ in range(n):
        outcome = 1 if rng.uniform(0, 1) < approval_rate else 0
        score = rng.normal(650, 50)
        rows.append((rid, outcome, gender, race, round(float(score), 1), "2026-01-15"))
        rid += 1

add_group(1200, "male", "white", 0.58)
add_group(1100, "female", "white", 0.42)
add_group(500, "male", "black", 0.40)
add_group(480, "female", "black", 0.36)
add_group(400, "male", "asian", 0.60)
add_group(380, "female", "asian", 0.54)
add_group(18, "nonbinary", "white", 0.50)     # below minimum sample size for gender x this cut
add_group(12, "male", "native_american", 0.45)  # below minimum sample size for race
add_group(40, None, "white", 0.50)            # NULL gender: excluded from the gender cut
add_group(35, "female", None, 0.45)           # NULL race: excluded from the race cut

cur.executemany("INSERT INTO applications VALUES (?, ?, ?, ?, ?, ?)", rows)
conn.commit()

MIN_SAMPLE = 30

query = f'''
WITH group_rates AS (
    SELECT 'gender' AS attribute, gender AS group_value,
           COUNT(*) AS n,
           SUM(CASE WHEN outcome = 1 THEN 1 ELSE 0 END) AS approvals,
           AVG(CASE WHEN outcome = 1 THEN 1.0 ELSE 0.0 END) AS approval_rate
    FROM applications
    WHERE gender IS NOT NULL
    GROUP BY gender

    UNION ALL

    SELECT 'race' AS attribute, race AS group_value,
           COUNT(*) AS n,
           SUM(CASE WHEN outcome = 1 THEN 1 ELSE 0 END) AS approvals,
           AVG(CASE WHEN outcome = 1 THEN 1.0 ELSE 0.0 END) AS approval_rate
    FROM applications
    WHERE race IS NOT NULL
    GROUP BY race
),
with_reference AS (
    SELECT *,
           MAX(approval_rate) OVER (PARTITION BY attribute) AS reference_rate
    FROM group_rates
)
SELECT
    attribute,
    group_value,
    n,
    approvals,
    ROUND(approval_rate, 4)              AS approval_rate,
    ROUND(reference_rate, 4)             AS reference_rate,
    CASE WHEN n >= {MIN_SAMPLE}
         THEN ROUND(approval_rate / reference_rate, 4)
         ELSE NULL
    END AS disparate_impact_ratio,
    CASE
        WHEN n < {MIN_SAMPLE} THEN 'INSUFFICIENT_SAMPLE'
        WHEN approval_rate / reference_rate < 0.8 THEN 'FLAGGED'
        ELSE 'OK'
    END AS flag
FROM with_reference
ORDER BY attribute, disparate_impact_ratio;
'''

cur.execute(query)
cols = [d[0] for d in cur.description]
print(" | ".join(f"{c:>20}" for c in cols))
for row in cur.fetchall():
    print(" | ".join(f"{str(v):>20}" for v in row))

# Sanity checks: confirm the null rows were excluded from both cuts.
cur.execute("SELECT COUNT(*) FROM applications WHERE gender IS NULL")
print("\nrows with NULL gender (excluded from the gender cut):", cur.fetchone()[0])
cur.execute("SELECT COUNT(*) FROM applications WHERE race IS NULL")
print("rows with NULL race (excluded from the race cut):", cur.fetchone()[0])
cur.execute("SELECT COUNT(*) FROM applications")
print("total rows in applications:", cur.fetchone()[0])

Executed output:

           attribute |          group_value |                    n |            approvals |        approval_rate |       reference_rate | disparate_impact_ratio |                 flag
              gender |            nonbinary |                   18 |                    6 |               0.3333 |               0.5294 |                 None |  INSUFFICIENT_SAMPLE
              gender |               female |                 1995 |                  844 |               0.4231 |               0.5294 |               0.7992 |              FLAGGED
              gender |                 male |                 2112 |                 1118 |               0.5294 |               0.5294 |                  1.0 |                   OK
                race |      native_american |                   12 |                    4 |               0.3333 |               0.5551 |                 None |  INSUFFICIENT_SAMPLE
                race |                black |                  980 |                  389 |               0.3969 |               0.5551 |                0.715 |              FLAGGED
                race |                white |                 2358 |                 1143 |               0.4847 |               0.5551 |               0.8732 |                   OK
                race |                asian |                  780 |                  433 |               0.5551 |               0.5551 |                  1.0 |                   OK

rows with NULL gender (excluded from the gender cut): 40
rows with NULL race (excluded from the race cut): 35
total rows in applications: 4165

The male group (approval rate 0.5294) is automatically the gender reference, since it has the highest rate, without that having been hardcoded anywhere in the query; female's ratio comes out to 0.7992, just under the 0.8 line, so it is correctly FLAGGED, and this is close enough to the threshold to illustrate why the exact reference-rate computation matters, a slightly different reference group choice could have pushed this either side of 0.8. On race, asian is the reference (0.5551), black is clearly flagged at 0.715, and white lands at 0.8732, above the line despite also being below the reference rate, showing the ratio, not just "is this group below the top group," is what the rule actually tests. Both the nonbinary gender group (n=18) and the native_american race group (n=12) are correctly reported as INSUFFICIENT_SAMPLE with a NULL ratio rather than a computed number, even though their raw approval rates (0.3333 for both) look dramatically low, which is exactly the case this safeguard exists for: a rate that looks alarming from 12 or 18 people is not yet evidence of anything. The 40 NULL-gender rows and 35 NULL-race rows are confirmed excluded from their respective cuts, and the total row count (4165) confirms nothing was silently dropped from the table itself, only from the specific aggregate where the attribute was missing.

Trade-offs and pitfalls

The most common mistake is hardcoding a single "majority" group as the denominator for every comparison rather than deriving the reference rate from the data, which silently changes the definition of the metric and can produce a different flag decision than the standard 80 percent rule would; if a business specifically wants a fixed historical reference group instead of the highest-rate group (sometimes required by a specific regulatory context), that choice should be an explicit, named parameter in the query, not an implicit accident of column ordering. A second pitfall is applying the 0.8 threshold to a group with a tiny sample without a floor, which either buries a real disparity in "not enough data to say" or manufactures a false alarm from three unlucky outcomes; the minimum-sample gate above is a coarse first line of defense, and a more rigorous version would attach a confidence interval to each group's rate and consider a flag confirmed only when the interval clearly excludes 0.8, not just when the point estimate does. A third is treating a passing disparate impact ratio as proof of no discrimination: the 80 percent rule is a screening heuristic for disparate IMPACT (an aggregate outcome pattern), not a test for disparate TREATMENT (a demonstrated intent or the use of a protected attribute or an obvious proxy for it directly in the decision process), and both are legally and practically distinct questions that this single query answers only the first of. Finally, running this query as a one-time report rather than a monitored metric misses the point: a ratio near but above 0.8 today, like female's 0.7992 here sitting a hair below it, or white's 0.8732 sitting closer to the line than the other passing groups, is a candidate for ongoing tracking, since a small shift in the underlying approval process could tip it over the threshold in either direction next period.

Direct answer: the query below groups approved transactions by gender, computes the approval rate for each group, and reports the absolute difference, treating missing (null) gender values by excluding them rather than folding them into either group.

Structured elaboration. The key correctness points: (1) SUM(approved) over a 0/1-encoded boolean column gives the count of approvals directly; (2) casting the numerator to a float before dividing avoids integer-division truncation to zero; (3) filtering WHERE gender IS NOT NULL before grouping, rather than after, ensures null-gender rows never silently form their own group or get miscounted into an existing one.

Worked example (executed against a small in-memory SQLite table).

Setup (a small, fully specified test table, so the query's output below is reproducible rather than merely asserted):

CREATE TABLE transactions (
    transaction_id INTEGER PRIMARY KEY,
    user_id INTEGER,
    amount REAL,
    approved INTEGER,
    country TEXT,
    gender TEXT,
    timestamp TEXT
);

INSERT INTO transactions VALUES
    (1, 1, 100.0, 1, 'US', 'female', '2026-01-01'),
    (2, 2, 200.0, 1, 'US', 'female', '2026-01-02'),
    (3, 3, 150.0, 0, 'US', 'female', '2026-01-03'),
    (4, 4, 300.0, 1, 'US', 'male',   '2026-01-04'),
    (5, 5, 250.0, 1, 'US', 'male',   '2026-01-05'),
    (6, 6, 400.0, 1, 'US', 'male',   '2026-01-06'),
    (7, 7, 120.0, 0, 'US', 'male',   '2026-01-07'),
    (8, 8, 90.0,  1, 'US', NULL,     '2026-01-08');

The query:

SELECT gender,
       SUM(approved) AS approved_count,
       COUNT(*) AS total_count,
       CAST(SUM(approved) AS FLOAT) / COUNT(*) AS approval_rate
FROM transactions
WHERE gender IS NOT NULL
GROUP BY gender

On this 8-row table (one row with a null gender, correctly excluded), the query returned:

female: approved_count=2, total_count=3, approval_rate=0.6667
male:   approved_count=3, total_count=4, approval_rate=0.75

Absolute difference: |0.6667 - 0.75| = 0.0833. The single null-gender row was correctly excluded from both groups' denominators rather than silently defaulting into one of them, which was verified by checking total_count sums to 7 (8 rows minus the 1 null row), not 8.

Trade-offs and pitfalls. (1) If gender is stored with inconsistent casing or trailing whitespace ("Female" vs "female "), the GROUP BY will silently create spurious extra groups; normalize with LOWER(TRIM(gender)) before grouping in a real production query. (2) On a very small table, this rate difference is not yet evidence of anything: report the sample size alongside the rate, and treat a small-sample gap as a prompt to gather more data or apply a statistical test, not as a conclusion on its own. (3) Do not report a rate at all for a group with a tiny denominator (say, fewer than 20-30 observations) without an explicit uncertainty caveat, since a difference computed from a handful of rows can flip sign with the next few transactions.