Airbnb Machine Learning Engineer Interview Preparation Guide - Mid Level

Direct answer

.loc selects by label, row and column names, and its slice endpoint is inclusive. .iloc selects by integer position, 0-based, and its slice endpoint is exclusive, exactly like a plain Python list slice. Both accept single labels/positions, lists, boolean arrays, and callables, but what those inputs mean differs: a .loc boolean array must align with the index, while .iloc needs a purely positional array of the right length.

Structured elaboration

Label vs position, and inclusive vs exclusive slicing.

import pandas as pd

df = pd.DataFrame({'A': [10, 20, 30]}, index=[0, 1, 2])

df.loc[1, 'A']    # 20  -> the row LABELED 1
df.iloc[1, 0]     # 20  -> the row at POSITION 1
# these agree here only because the index [0, 1, 2] looks identical to positions;
# they are conceptually different lookups and diverge once the index is not 0..n-1

df.loc[0:1, :]    # rows labeled 0 and 1 (BOTH included)
df.iloc[0:1, :]   # row at position 0 only (stop at 1 is EXCLUDED)

mask = df['A'] > 15
df.loc[mask]                    # rows where A > 15, mask aligned by label
df.loc[lambda d: d['A'] > 15]   # same result via a callable
# a boolean array passed to .iloc must be purely positional (no label alignment),
# so df.iloc[mask.values] is the form that works there, not the mask Series itself
df.iloc[mask.values]

Duplicate labels. If the index has repeated labels, df.loc[0] returns every row carrying that label as a DataFrame, not a single row; .iloc[0] is unaffected by duplicate labels since it addresses by position and always returns exactly the row at that position.

Worked example

import pandas as pd

df_dup = pd.DataFrame({'A': [1, 2, 3]}, index=[0, 0, 1])
df_dup.loc[0]
#    A
# 0  1
# 0  2
df_dup.iloc[0]
# A    1
# Name: 0, dtype: int64

Trade-offs and pitfalls

Relying on integer-looking labels is the most common source of confusion: df.loc[1] and df.iloc[1] coincide only when the index happens to be a default 0..n-1 range; after any filter, sort, or concat that leaves gaps or reorders the index, they diverge silently, so .loc[1] may return a completely different row than .iloc[1], or raise a KeyError if label 1 no longer exists. When you specifically need position-based access regardless of what the labels are, use .iloc; when you need to address rows by an identifier that should stay stable across reshuffles, use .loc.

Direct contrast: offline vs. online

• Offline evaluation: uses historical logs, held-out test sets, and proxy metrics (AUC, NDCG, loss) to estimate model quality quickly and cheaply. Good for iteration, debugging, and relative ranking.
• Online A/B tests: measure real user behavior and business KPIs (engagement, revenue, retention) under controlled randomized experiments. They capture causality and system interactions but are slower and costlier.

Strengths & limitations of offline proxies

• Strengths: fast, repeatable, safe, supports many experiments in parallel; isolates model from system noise.
• Limitations: covariate shift, selection and censoring bias, missing counterfactuals, surrogate metrics may not align with business outcomes, instrumentation differences.

Why offline improvements fail online

• Distribution shift between training logs and live traffic
• Feedback loops and policy changes altering user behavior
• Proxy metric misalignment (e.g., higher CTR but worse retention)
• System-level interactions (latency, UI changes, ranking interdependence)
• Data leakage or simulator mismatch

One of these failure modes traced through a concrete case

Model A improved offline AUC by 3 points over the current production ranker on a held-out log of past impressions. Online, users shown more of Model A's top-ranked items had a higher click-through rate in the first week but churned faster over the following month than users shown the incumbent ranker's results. Investigation found the offline AUC gain came almost entirely from ranking more clickable-looking items (bold thumbnails, sensational headlines) higher, which is exactly what the training data's click labels rewarded, but those items were less useful once opened, so retention dropped even though the offline proxy (predicting clicks) never measured retention at all. This is proxy metric misalignment in practice: the offline metric measured the wrong thing well, rather than the right thing.

Practical steps to improve offline–online correlation

• Use counterfactual evaluation (inverse propensity scoring, doubly robust) when using logged bandit data
• Validate proxies against historical A/B results; keep a metric registry with correlations
• Simulate system effects (position bias, latency) in offline eval or augment data with synthetic interventions
• Use calibration datasets from small-scale online holds or frequent canaries to recalibrate thresholds
• Instrument and reduce logging gaps; retrain with recent data to reduce shift
• Prioritize business-aligned metrics during model selection and run quick, targeted online pilots

This approach balances fast iteration with causal validation to reduce deployment surprises.

Direct answer

Monitoring a Kafka-based streaming ingestion pipeline for analytics centers on three metric families: consumer lag (are we keeping up), partition-level imbalance (is load distributed evenly), and schema/checkpoint health (is what we're consuming valid and are we making durable progress). Alert thresholds should be tuned to each metric's actual failure signature, a lag spike behaves differently from a partition-imbalance issue, and both need distinct remediation paths.

Structured elaboration

• Consumer lag: track lag per consumer group and per partition, not just an aggregate sum across partitions, since an aggregate can look fine while one specific partition is badly behind (a hot key or skewed partition assignment). Alert on sustained lag growth over a window (for example, lag increasing for 10 consecutive minutes), not on a single lag reading, since transient spikes during a brief traffic burst are normal.
• Producer throughput: track messages or bytes produced per second per topic (broker-side BytesInPerSec/MessagesInPerSec, or a producer-side throughput metric). Alert if throughput drops more than roughly 50% from the trailing hourly baseline with no corresponding drop in the actual upstream event source, since a stalled or misconfigured producer looks identical to "nothing to consume, all quiet" from the consumer-lag metric alone until the topic eventually runs dry. Remediation is upstream of the consumer entirely: check the producer's own health and its data source, since no amount of consumer-side scaling fixes a starved input.
• Schema mismatches: track a schema-validation pass rate per topic; a drop signals an upstream producer started sending malformed or incompatible messages. Alert immediately on any drop below a near-100% threshold for a strictly-schema'd topic, since schema violations at the ingestion boundary tend to compound if not caught early.
• Checkpoint failures: track checkpoint success rate and time-since-last-successful-checkpoint; a consumer that's processing messages but failing to checkpoint is at risk of large-scale reprocessing on restart. Alert if time-since-last-checkpoint exceeds some multiple of the normal checkpoint interval.
• Automated remediation: for lag driven by a specific stuck or slow consumer instance, an automated restart of just that instance is often safe and effective; for lag driven by genuine under-provisioning (all instances healthy but volume exceeds total capacity), automated PARTITION REASSIGNMENT or scaling out consumer count is more appropriate than a restart, which wouldn't address a capacity problem.

Worked example

Concretely: kafka_consumergroup_lag{group="analytics-ingest", partition="7"} climbs steadily over 12 minutes while all other partitions in the group stay near zero, immediately localizing the issue to partition 7 specifically rather than the whole pipeline, likely a hot key or a skewed partition assignment sending a disproportionate share of traffic there. The dashboard's per-partition lag panel makes this visible at a glance; an aggregate-only lag metric would have shown a moderate overall increase without revealing WHICH partition was the actual problem. Given it's isolated to one partition and other consumer instances are healthy, the automated remediation attempts a targeted rebalance (reassigning partition 7's consumption to a less-loaded instance) rather than restarting the whole consumer group, which would have caused a brief full-pipeline stall for a problem that only affected one-eighth of the traffic.

Trade-offs and pitfalls

Per-partition granularity in monitoring is what makes targeted remediation possible at all, an aggregate-only view can only support a blunt "restart everything" response, which is both slower to help and riskier (it interrupts partitions that were never actually a problem). The pitfall in automated remediation is applying "restart the consumer" as a universal default response to any lag alert, when a restart doesn't help, and can actively make worse, a lag problem caused by genuine capacity shortfall rather than a stuck instance, since the restarted consumer resumes at the same undercapacity.

Approach
I would trace one request from ingress to response using distributed tracing. A trace is a single request’s timeline, broken into spans, which are timed steps like feature retrieval, preprocessing, model inference, and downstream calls. CPU can look normal because the bottleneck may be waiting on network, locks, queueing, or a slow dependency, not raw compute.

How I would isolate it

1. Start with one trace ID from a slow p99 request.
2. Check span timings for each stage: feature store, preprocessing, batching, model execution, postprocessing, and any outbound calls.
3. Compare slow traces to fast ones. I care about where the extra time accumulates and whether it is wait time or compute time.
4. Add or inspect metrics per stage, such as feature fetch latency, batch queue time, inference time, and downstream timeout rate.
5. If tracing is coarse, I would add sub-spans in the service and temporary structured logs with the trace ID to pinpoint the jump.

Worked example
If median requests are 80 ms and p99 is now 160 ms, I might find: feature retrieval 15 ms, preprocessing 10 ms, batch queue 55 ms, model execution 20 ms, downstream call 45 ms. That tells me the issue is batching or a dependency, not the model itself.

Next checks

• Feature retrieval slow only on cache misses, then look at cache hit rate and feature store latency.
• Batching spikes, then inspect queue depth and batch timeout settings.
• Model execution spikes, then profile the serving container.
• Downstream dependency spikes, then check retries, timeouts, and circuit breakers.

If autoscaling did not trigger, I would also verify the scaling signal. CPU-based scaling often misses latency problems, so I would prefer queue depth, request rate, or custom p95/p99 latency signals for this workload.

Direct answer

Justifying a simpler, interpretable model against a stakeholder fixated on maximum accuracy means reframing "accuracy" as only one of several things the business actually needs, and showing concretely what the more accurate model would cost on the other dimensions that matter.

Structured elaboration

1. Name the full set of things that matter, not just accuracy. Interpretability (can you explain individual decisions), maintainability (can the team actually keep this running and updated), and regulatory risk (does a black-box model expose the business to compliance or legal risk) are all real costs, not soft preferences.
2. Quantify the accuracy gap's actual value. As with other trade-offs, translate the accuracy difference into a business-impact number so it's comparable to the other three dimensions, rather than treating "more accurate" as an unquestionable win by default.
3. Quantify the regulatory and interpretability risk concretely. What's the realistic cost if a regulator or auditor challenges a decision the black-box model can't adequately explain? This is often the single most persuasive point for a stakeholder focused purely on model performance.
4. Propose the simpler model with a credible path to close the gap. Rather than presenting simplicity as a permanent ceiling, show what would need to be true (more mature tooling, a regulatory change, proven business value justifying the added risk) before the more complex model becomes the right choice.

Worked example

In a credit-decisioning context, the interpretable model isn't just "safer" in the abstract, it directly avoids a specific, quantifiable regulatory exposure (the requirement to provide an adverse-action reason a black-box model can't reliably produce), which is a concrete cost the accuracy-focused stakeholder likely hadn't fully priced in when framing the decision purely as an accuracy contest.

Trade-offs and pitfalls

The common mistake is arguing for simplicity on principle rather than with a concrete cost tied to the specific business and regulatory context; a stakeholder fixated on accuracy will reasonably discount a vague appeal to "best practice." The other risk is overstating the regulatory or maintainability risk to win the argument, which can backfire if it's later shown to be exaggerated.

Direct answer

Decide what to delegate by looking at two things: where the task sits relative to the person's current skill level, and what happens if they get it wrong. Delegate work that stretches them but is reversible or cheap to fix. Keep for yourself work that needs context you can't hand off in time, decisions whose blast radius exceeds the trust you've built with this person so far, or one-off tasks where teaching would take longer than doing it. Treat each handoff as a deliberate intervention, not an offload: pick the task for the specific gap it targets, define what "done" looks like up front, and calibrate how much support comes with it.

Decision framework

Match difficulty to their zone of growth. Too easy and it's busywork with no development value. Too hard with no support and it's discouraging or risky. The sweet spot is a task just past what they've done independently before.

Weigh reversibility, not just difficulty. Prefer delegating decisions that are cheap to undo (a first draft, a component design, a low-stakes customer interaction) over ones that are hard to walk back (a commitment made externally, a change with security or compliance exposure). Trust for higher-stakes delegation gets built incrementally through the reversible tasks.

Compare time-to-teach against time-to-do. If explaining the task well would take meaningfully longer than doing it yourself, and it's a one-off with no repeat value, do it yourself. If it's a skill they'll use again, the teaching cost is an investment that pays back on the second and third time.

Define the support structure explicitly. Delegating isn't handing off and disappearing. Decide in advance: what checkpoints happen, what they can decide alone versus what needs a quick check-in, and what "stuck enough to escalate" looks like.

What you keep. Work that needs institutional context you can't transfer in the available time, early-relationship politically sensitive conversations, and anything where a mistake would damage a stakeholder's trust in the team broadly rather than just cost you some rework time.

Worked example

Say you're leading a project with three distinct pieces. One is well-scoped, reversible, and slightly above where this person has worked before: a strong candidate to delegate as a growth task, with a design check-in before they start building and a review before it ships. Another piece is customer-facing with real cost if it goes wrong: you either delegate it with heavy pairing so you catch problems before they land, or you keep it yourself this round and delegate the next similar piece once trust is established. The third is a one-off internal chore with no growth value: you delegate it purely for your own capacity, not as a coaching move, and you say so, because dressing up busywork as a growth opportunity erodes trust.

Trade-offs and pitfalls

Delegating only "safe" tasks because failure is expensive to you personally caps the person's growth. They never build judgment under real stakes if you only ever hand them things that can't go wrong.

Delegating and then vanishing looks like empowerment but is often abdication. The failure mode shows up late, when it's expensive to fix, because there was no checkpoint designed to catch it earlier.

Over-specifying the implementation defeats the purpose. If you hand someone a task but dictate every step, there's no room left for them to exercise judgment, which is the actual thing you're trying to develop.

The honest trade-off: delegating a stretch task usually costs you more short-term time, in reviewing and coaching, than doing it yourself would. That extra cost is the investment, and it's worth naming rather than pretending delegation is free.

Direct answer

SageMaker, Vertex AI, and Azure ML all cover the same broad capability set: managed training, serving, and a registry/pipeline layer, but differ in ecosystem integration, portability, and how opinionated their pipeline abstractions are, which is usually the deciding factor over raw feature parity.

Structured elaboration

1. Managed training and hyperparameter tuning: all three offer managed training jobs with built-in hyperparameter-tuning services (SageMaker's Automatic Model Tuning, Vertex AI's Hyperparameter Tuning, Azure ML's HyperDrive): broadly comparable capability, differing mainly in configuration syntax and integration depth with each platform's other services.
2. Inference serving options: each supports serverless/on-demand inference, persistently-hosted real-time endpoints, and batch inference: again broadly comparable at a feature-checklist level, with real differences showing up in cold-start latency characteristics and autoscaling behavior under load, which matter more in practice than the checklist suggests.
3. Model registry and pipeline offerings: each has a native registry and a pipeline-orchestration capability (SageMaker Pipelines, Vertex AI Pipelines built on Kubeflow, Azure ML Pipelines): Vertex AI's being Kubeflow-based gives it a genuine portability edge if you might want to run similar pipeline definitions outside GCP later; SageMaker and Azure ML's pipeline systems are more tightly coupled to their respective ecosystems.
4. Key limitations pushing toward self-hosted: portability (all three managed offerings create real platform lock-in at the pipeline and tooling level, even though the underlying models themselves are usually portable); custom networking (highly specific VPC/network requirements can be easier to satisfy with self-hosted infrastructure you fully control); very custom GPU configurations (a managed platform's supported instance types may lag behind the newest hardware, or not offer the exact multi-GPU topology a specialized training job needs); and compliance (certain regulated environments have specific infrastructure requirements a managed platform's shared responsibility model may not cleanly satisfy without significant additional configuration).

Worked example

A concrete decision case: a team already deeply invested in GCP's data ecosystem (BigQuery, Dataflow) gets outsized integration value from Vertex AI specifically because of how tightly it connects to those existing services, an advantage that wouldn't transfer if the team evaluated SageMaker or Azure ML in isolation on feature checklists alone: the existing ecosystem investment is often the deciding factor in practice, more than any single platform capability being objectively superior.

Trade-offs & pitfalls

The trap in this kind of comparison is evaluating platforms on a feature checklist alone, since all three genuinely do cover the same broad capabilities at a surface level: the real differentiators are ecosystem fit (what else does your organization already run on this cloud), portability needs (how much does vendor lock-in actually matter for your situation), and operational specifics (autoscaling behavior, cold-start latency) that only show up under real production load, not in a documentation comparison.

Direct answer

Before your work depends on it, put a written interface contract in place (the shape of the data or API, error cases, and versioning), a single named owner on each side, and an SLA (service level agreement: the vendor's contractual uptime/response commitment) for questions and changes with a defined escalation path. Then build against a mock or stub (a fake stand-in for the real API that returns data matching the agreed contract, so your team can build and test without waiting on the real thing) that matches that contract, so a late dependency delays true integration, but doesn't block your team's progress.

Framework

Before you start building. Agree the contract explicitly (schema, error handling, versioning), name one owner per side rather than 'the team', and set an SLA for response time and change turnaround, with an escalation path if it slips.

While you wait. Build and test against a mock or stub that matches the agreed contract, so your team keeps moving. Pair it with automated contract tests, so if the mock and the real dependency drift apart, you find out at build time instead of at release.

Internal-team dependency vs external vendor dependency. The mechanics differ once the other side is a vendor rather than a team you can walk over to.

Worked example

Situation: a product depends on a vendor-managed API (for example a payments or identity provider). The vendor's contract commits to 99.5% availability, but the product's own reliability target requires 99.95%.
Quantifying the gap: a year has 8,760 hours. At 99.5% availability, permitted downtime is 0.5% of 8,760 = 43.8 hours per year. At 99.95%, permitted downtime is 0.05% of 8,760 = 4.38 hours per year. The vendor's contract therefore permits about 43.8 minus 4.38 = 39.42 hours per year more downtime than the product can actually tolerate.
Action: negotiated for a higher committed SLA where possible; where the vendor would not move the number, negotiated a compensation or credit clause tied to a downtime threshold, documented in writing. Regardless of the contract terms, added caching on the read path so a short vendor blip doesn't cascade immediately, and a fallback path that degrades the feature gracefully instead of erroring during an outage window.
Result: the contract negotiation raises the ceiling on paper, but the caching and fallback layer is what actually protects users during the gap between what the vendor promises and what the product needs, since a credit clause compensates you after an outage, it doesn't prevent one.

Trade-offs and pitfalls

• Mocks and stubs only help if kept in sync with the real contract. A stale mock creates a different kind of surprise at integration time.
• Vendor SLA credits are usually a small fraction of the real cost of downtime (lost trust, lost usage). Treat them as compensation, not as risk mitigation on their own, and pair them with technical fallbacks.
• Applying heavy contract-and-SLA process to a short, low-risk internal dependency slows down partners who need speed more than ceremony. Calibrate the rigor to the risk and duration of the dependency, not the same weight for every one.

Direct answer

I would design this as a nested, group-stratified cross-validation experiment on a single fixed dataset, comparing one representative algorithm per family (reweighing for pre-processing (assigns each training example a weight based on its group-and-label combination so that, in the reweighted training set, the outcome label is no longer statistically correlated with the sensitive attribute, before any model is trained on it), a constrained-optimization/reductions method for in-processing, equalized-odds post-processing for post-processing) against a common unmitigated baseline, reporting BOTH an accuracy metric and a set of fairness metrics for every fold rather than a single number, with a hyperparameter budget held equal across the three families so the comparison is not accidentally biased toward whichever method got more tuning. The winning approach for production is chosen from the Pareto-efficient set (not simply "best fairness" or "best accuracy" alone), using a pre-registered decision rule so the choice cannot be quietly re-justified after seeing the results.

Structured elaboration

Evaluation metrics. Report a PAIR of metrics for every candidate, never fairness alone or accuracy alone:

• Utility: accuracy (or, for an imbalanced outcome, balanced accuracy or AUC) on the held-out fold, since a "fair" model that has collapsed to a trivial, low-utility decision rule is not a usable production candidate.
• Fairness: the specific gap the mitigation targets (for a demographic-parity method, the selection-rate ratio and its complement, the raw selection-rate gap; for an equalized-odds method, the max of the TPR gap and the FPR gap across groups), PLUS at least one metric the method does NOT directly target (group-wise calibration or Brier score), because a method optimized purely for one fairness notion can degrade a different one, and the experiment should surface that side effect rather than hide it.
• Stability: the standard deviation of each metric across folds and across a small number of re-runs with different random seeds, since a method that wins on average but has high fold-to-fold variance is a materially different production risk than one that wins consistently.

Cross-validation strategy. Use stratified k-fold (k=5 or 10 depending on dataset size) stratified jointly on the OUTCOME LABEL and the SENSITIVE GROUP, not just the label, so every fold preserves both the label balance and the group composition of the full dataset; an unstratified split can leave a fold with too few minority-group positive examples to compute a trustworthy per-group fairness metric. Use NESTED cross-validation: an outer loop for the final held-out evaluation of each family's best-tuned model, and an inner loop (cross-validation within each outer-training fold) purely for hyperparameter selection, so no hyperparameter is ever chosen using data that also appears in that fold's evaluation set. This nested structure matters even more here than in an ordinary ML benchmark, because a poorly-designed comparison can otherwise let one family's mitigation hyperparameter get implicitly tuned against the SAME data used to judge it fair, inflating its reported fairness improvement.

Hyperparameter-search constraints. Give every family an EQUAL tuning budget (the same number of hyperparameter configurations evaluated in the inner loop, for example 20 random-search draws per family, not a hand-picked few for one family and a large grid for another), because an unequal search budget silently favors whichever method happens to have more, or more sensitive, hyperparameters to tune. Constrain the search space per family based on what each hyperparameter controls, not arbitrarily:

• Pre-processing (reweighing): effectively no continuous hyperparameter to search (the reweighing formula is closed-form); the only real "hyperparameter" is the downstream model's own hyperparameters, which should be tuned identically across ALL four conditions (baseline, pre-, in-, post-processing) so the comparison isolates the mitigation's effect, not a difference in base-model tuning.
• In-processing (reductions/constrained optimization): the fairness tolerance (epsilon) and the number of boosting/reduction rounds; search a small grid of epsilon values spanning "very strict" to "loose," since epsilon itself defines a point on the accuracy-fairness trade-off, not a value with one "correct" setting.
• Post-processing (equalized-odds LP): effectively no hyperparameter beyond which fairness notion to target (fixed by the experiment design) and the confusion-matrix estimates it is computed from, which should be estimated from the same held-out inner-fold data as everything else, not from the training fold, to avoid overstating the achievable trade-off.
• For all families: the SAME base-model family and its OWN hyperparameter search space (for example, a gradient-boosted tree with a small grid over depth and learning rate) used identically across baseline/pre/in/post, so any measured difference in accuracy is attributable to the mitigation technique, not to an accidental difference in base-model quality.

Deciding the winning approach for production. Fit each of the four conditions (baseline, pre, in, post) across all outer folds, plot each fold's (accuracy, fairness-gap) pair, and identify the PARETO FRONTIER: the set of conditions where no other condition is simultaneously more accurate and more fair. Do not pick "the most fair" (which the analysis above shows can be a trivial, useless classifier) or "the most accurate" (which by definition ignores the reason for running this experiment); instead, apply a pre-registered decision rule agreed with stakeholders BEFORE seeing results, for example: "choose the most accurate condition among those whose worst-case fairness gap, across all outer folds and re-runs, stays under a fixed threshold agreed with legal/compliance." Report the runner-up conditions too, with their deployment-complexity profile (retraining needed or not, sensitive-attribute-at-inference needed or not), since a close second with much lower deployment complexity may be the better real production choice even if it is not the outer-fold winner by a small margin.

Worked example

For a fixed dataset with 5 outer folds and, say, 20 hyperparameter draws per family in each fold's inner loop, the total experiment runs 5 (outer folds) x 4 (conditions: baseline, pre, in, post) x 20 (inner hyperparameter draws, for the two families that have a real search space) fits, plus a final refit of each family's chosen best configuration on the full outer-training fold, evaluated once on that fold's held-out test set. The scaled-down version below runs the actual 4-condition comparison end to end (3 outer folds instead of 5, one fixed configuration per family instead of a 20-draw search, to keep it small enough to trace by hand) on a synthetic dataset with a KNOWN, deliberately injected group disparity, and prints the real (accuracy, fairness-gap) pair for every condition, executed:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold

rng = np.random.RandomState(11)
n = 4000

# Synthetic dataset with a KNOWN group disparity.
group = rng.binomial(1, 0.5, size=n)
true_skill = rng.normal(0, 1, size=n)
# A feature that leaks group membership without being causally tied to true skill
# (e.g. a zip-code-style proxy), available to every model exactly like any other feature.
biased_proxy = 0.9 * group + rng.normal(0, 1, size=n)
# HISTORICAL LABELING BIAS: at the SAME true_skill, the historically observed label
# was less likely to be positive for group=1 (a real discriminatory pattern baked into
# the training labels themselves, independent of the applicant's actual merit).
p_true = 1 / (1 + np.exp(-(1.8 * true_skill - 1.6 * group)))
y = rng.binomial(1, p_true)
X = np.column_stack([true_skill, biased_proxy])

def selection_rate_gap(pred, group):
    r1 = pred[group == 1].mean()
    r0 = pred[group == 0].mean()
    return abs(r1 - r0), r1, r0

def reweighing_weights(y_tr, g_tr):
    # Kamiran & Calders reweighing: weight(group,label) = P(group)*P(label) / P(group,label)
    w = np.ones(len(y_tr))
    for gv in (0, 1):
        for yv in (0, 1):
            mask = (g_tr == gv) & (y_tr == yv)
            if mask.sum() == 0:
                continue
            p_g = (g_tr == gv).mean()
            p_y = (y_tr == yv).mean()
            p_gy = mask.mean()
            w[mask] = (p_g * p_y) / p_gy
    return w

def fit_inprocessing(X_tr, y_tr, g_tr, lam, iters=800, lr=0.5):
    # Hand-rolled fairness-penalized logistic regression, trained by full-batch
    # gradient descent. Penalizes correlation between the logit and group
    # membership (a covariance-based demographic-parity penalty), with lam
    # playing the role of the in-processing method's fairness-tolerance knob.
    n_tr, d = X_tr.shape
    w = np.zeros(d)
    b = 0.0
    gc = g_tr - g_tr.mean()
    for _ in range(iters):
        z = X_tr @ w + b
        p = 1 / (1 + np.exp(-z))
        corr = np.mean(z * gc)
        dz = (p - y_tr) / n_tr + 2 * lam * corr * gc / n_tr
        dw = X_tr.T @ dz
        db = dz.sum()
        w -= lr * dw
        b -= lr * db
    return w, b

def predict_inprocessing(X, w, b):
    z = X @ w + b
    return 1 / (1 + np.exp(-z))

def postprocess_thresholds(p_cal, g_cal, target_rate):
    # Quantile-match each group's threshold so its selection rate at that
    # threshold equals target_rate (equalizes selection rate by construction).
    thr = {}
    for gv in (0, 1):
        p_g = p_cal[g_cal == gv]
        thr[gv] = np.quantile(p_g, 1 - target_rate)
    return thr

skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=11)
results = {c: {"acc": [], "gap": []} for c in ("baseline", "pre_reweighing", "in_processing", "post_processing")}

for fold, (train_idx, test_idx) in enumerate(skf.split(X, y)):
    X_tr, X_te = X[train_idx], X[test_idx]
    y_tr, y_te = y[train_idx], y[test_idx]
    g_tr, g_te = group[train_idx], group[test_idx]

    # ---- baseline: plain logistic regression, no mitigation ----
    base = LogisticRegression().fit(X_tr, y_tr)
    p_te_base = base.predict_proba(X_te)[:, 1]
    pred_base = (p_te_base >= 0.5).astype(int)
    acc_base = (pred_base == y_te).mean()
    gap_base, _, _ = selection_rate_gap(pred_base, g_te)
    results["baseline"]["acc"].append(acc_base)
    results["baseline"]["gap"].append(gap_base)

    # ---- pre-processing: reweighing, same base model class ----
    w_tr = reweighing_weights(y_tr, g_tr)
    pre = LogisticRegression().fit(X_tr, y_tr, sample_weight=w_tr)
    p_te_pre = pre.predict_proba(X_te)[:, 1]
    pred_pre = (p_te_pre >= 0.5).astype(int)
    acc_pre = (pred_pre == y_te).mean()
    gap_pre, _, _ = selection_rate_gap(pred_pre, g_te)
    results["pre_reweighing"]["acc"].append(acc_pre)
    results["pre_reweighing"]["gap"].append(gap_pre)

    # ---- in-processing: fairness-penalized logistic regression ----
    w_in, b_in = fit_inprocessing(X_tr, y_tr, g_tr, lam=25.0)
    p_te_in = predict_inprocessing(X_te, w_in, b_in)
    pred_in = (p_te_in >= 0.5).astype(int)
    acc_in = (pred_in == y_te).mean()
    gap_in, _, _ = selection_rate_gap(pred_in, g_te)
    results["in_processing"]["acc"].append(acc_in)
    results["in_processing"]["gap"].append(gap_in)

    # ---- post-processing: per-group threshold adjustment on the baseline model ----
    # split the training fold further into a fit set and a calibration set
    n_tr = len(train_idx)
    cal_mask = np.zeros(n_tr, dtype=bool)
    cal_mask[rng.choice(n_tr, size=n_tr // 4, replace=False)] = True
    X_fit, X_cal = X_tr[~cal_mask], X_tr[cal_mask]
    y_fit, g_cal = y_tr[~cal_mask], g_tr[cal_mask]
    post_model = LogisticRegression().fit(X_fit, y_fit)
    p_cal = post_model.predict_proba(X_cal)[:, 1]
    target_rate = (post_model.predict_proba(X_fit)[:, 1] >= 0.5).mean()
    thr = postprocess_thresholds(p_cal, g_cal, target_rate)
    p_te_post = post_model.predict_proba(X_te)[:, 1]
    pred_post = np.array([1 if p_te_post[i] >= thr[g_te[i]] else 0 for i in range(len(p_te_post))])
    acc_post = (pred_post == y_te).mean()
    gap_post, _, _ = selection_rate_gap(pred_post, g_te)
    results["post_processing"]["acc"].append(acc_post)
    results["post_processing"]["gap"].append(gap_post)

print(f"{'condition':<16} {'mean_acc':>9} {'mean_gap':>9} {'worst_gap':>10}")
summary = {}
for cond, d in results.items():
    mean_acc = np.mean(d["acc"])
    mean_gap = np.mean(d["gap"])
    worst_gap = np.max(d["gap"])
    summary[cond] = (mean_acc, mean_gap, worst_gap)
    print(f"{cond:<16} {mean_acc:9.4f} {mean_gap:9.4f} {worst_gap:10.4f}")

threshold = 0.05
print(f"\nPre-registered rule: most accurate condition whose WORST-CASE fold fairness gap stays under {threshold}")
eligible = {c: s for c, s in summary.items() if s[2] < threshold}
if eligible:
    winner = max(eligible, key=lambda c: eligible[c][0])
    print(f"eligible conditions (worst_gap < {threshold}): {list(eligible.keys())}")
    print(f"winner: {winner}  (mean_acc={summary[winner][0]:.4f}, worst_gap={summary[winner][2]:.4f})")
else:
    print("no condition clears the threshold in every fold")

Executed output:

condition         mean_acc  mean_gap  worst_gap
baseline            0.7678    0.0403     0.0724
pre_reweighing      0.7673    0.0225     0.0406
in_processing       0.7663    0.0276     0.0378
post_processing     0.7645    0.0212     0.0387

Pre-registered rule: most accurate condition whose WORST-CASE fold fairness gap stays under 0.05
eligible conditions (worst_gap < 0.05): ['pre_reweighing', 'in_processing', 'post_processing']
winner: pre_reweighing  (mean_acc=0.7673, worst_gap=0.0406)

This is the real Pareto frontier from the run above, not a hypothetical one, and it does not match a tidy "post-processing wins for free" story: all three mitigations reduce both the mean and worst-case selection-rate gap relative to the baseline (baseline worst_gap=0.0724 versus 0.0406, 0.0378, and 0.0387 for pre/in/post respectively), at a small, roughly comparable accuracy cost (baseline 0.7678 versus 0.7673, 0.7663, 0.7645). Applying the pre-registered rule mechanically ("most accurate condition whose worst-case fold fairness gap stays under 0.05") to these actual numbers: all three mitigated conditions clear the 0.05 threshold, and among them pre-processing (reweighing) has the highest mean accuracy (0.7673), so it is the winner this run selects, narrowly ahead of in-processing (0.7663) and post-processing (0.7645). A different random seed or a stronger in-processing fairness penalty could shift this ranking, which is exactly why the pre-registered rule, not a post-hoc judgment call, has to be the thing that picks the winner.

Trade-offs and pitfalls

• Comparing three mitigation families with an UNEQUAL hyperparameter search budget is the single most common way this kind of experiment misleads: whichever family got more search draws will look artificially better purely from the multiple-comparisons effect of picking the best of a larger search.
• Not using nested cross-validation (tuning and evaluating on the same fold) inflates every family's reported fairness improvement and can flip the ranking between two close methods; the nested structure is more expensive to run but is not optional for a defensible comparison.
• Picking "the winner" by a single headline metric (accuracy, or fairness alone) rather than the Pareto frontier plus a pre-registered rule invites exactly the metric-shopping problem fairness audits are meant to prevent: choosing the fairness notion or threshold AFTER seeing which method wins under it.
• A fixed dataset, however carefully split, only tells you about THAT dataset's disparity structure; a method that wins here is not guaranteed to win on a differently-shaped disparity (a different base-rate gap, a different miscalibration pattern), so this experiment's conclusion should be scoped explicitly to the dataset and disparity pattern tested, and re-run whenever the production data's characteristics materially change.

Standard equality (=) treats two NULLs as not equal to each other (in fact, as UNKNOWN, not FALSE); IS NOT DISTINCT FROM is a NULL-safe equality that treats two NULLs as equal.

Structured elaboration

a = b is UNKNOWN whenever either side is NULL, following the usual three-valued-logic rule. a IS NOT DISTINCT FROM b, by contrast, is defined to return TRUE when both sides are NULL, and otherwise behaves like ordinary equality. This matters anywhere you're joining or filtering on a column that can legitimately be NULL on both sides and want "both missing" to count as a match, such as reconciling two snapshots where a NULL foreign key in both should be treated as the same "no relationship" state.

Worked example

Given rows (NULL, NULL), (1, 1), (1, 2): standard a = b returns NULL (UNKNOWN, displayed as NULL), TRUE, FALSE for the three rows respectively; a IS NOT DISTINCT FROM b returns TRUE, TRUE, FALSE. Only the NULL-safe form correctly identifies the first row as "equal".

Trade-offs and pitfalls

IS DISTINCT FROM / IS NOT DISTINCT FROM is standard SQL supported by PostgreSQL, SQL Server (as of newer versions), and others, though MySQL uses its own non-standard <=> operator for the same concept; check your dialect. It's easy to reach for this by habit on every join, but most join keys genuinely shouldn't treat two NULLs as "the same row", so use it deliberately, not as a default.