Amazon Machine Learning Engineer Interview Preparation Guide (Mid-Level)

Direct answer: An interaction feature is the product (or another combination) of two existing features, and a polynomial feature raises a single feature to a higher power; both aim to let a model that can only represent additive, linear relationships capture a joint or curved effect it otherwise couldn't, but both can also actively hurt generalization if applied without restraint.

Structured elaboration:
An interaction term (like price times discount rate) helps when the TRUE relationship genuinely depends on the combination of two variables together, not either alone: a discount's effect on purchase likelihood might be completely different at a high price point than a low one, a relationship a purely additive model (with price and discount as separate, unrelated inputs) structurally cannot represent. A high-degree polynomial feature can similarly let a model fit a curved relationship a linear model alone would miss.

Worked example: Adding price × discount_rate as an explicit feature lets a linear model represent that a 20% discount matters very differently for a $10 item versus a $10,000 item, an effect the two features alone (fed in separately) cannot capture no matter how they're weighted individually. Conversely, adding high-degree polynomial terms (price squared, price cubed, up through a high order) on a modest dataset commonly causes overfitting: the model fits increasingly wiggly curves that chase noise in the training data rather than the true underlying relationship, and generalizes poorly to new data specifically because there wasn't enough data to reliably estimate all those additional polynomial coefficients.

Trade-offs and pitfalls: The failure mode in both cases is the same at its root: adding flexibility (whether via an interaction or a higher-degree polynomial) without enough data to support estimating the added parameters reliably trades a small chance of capturing real additional signal against a much larger risk of fitting noise; regularization, feature selection after expansion, and simply testing whether the added terms actually help on a genuine holdout (rather than assuming more flexibility is free) are the standard guardrails.

Direct answer

Keep alignment across time zones with three levers: shrink what actually needs real-time overlap by defaulting to async updates on a fixed template, protect a small deliberately scheduled overlap window for anything that truly needs live discussion, and make handoffs explicit in writing so context transfers cleanly across the boundary instead of depending on someone's memory.

Framework

Reduce dependence on overlap. Default to async status updates on a fixed cadence, and use written decision docs rather than requiring a live meeting for every decision. Most updates don't need a room, only genuinely ambiguous or high-stakes calls do.

Protect a deliberate overlap window. Negotiate a recurring block, even a short one, and rotate who takes the inconvenient time so the burden doesn't always fall on the same region.

Make handoffs explicit. When work crosses a time-zone boundary, produce a short written artifact rather than relying on a quick chat message. This matters most in ops-heavy, always-on contexts.

Worked example

Consider an on-call rotation providing 24/7 production coverage across three time zones (for example [Region A], [Region B], and [Region C]), where the two outer regions have little or no live overlap with each other.

• Shadow and overlap periods: the incoming region's on-call shadows the outgoing region's on-call for a short deliberate window at the shift boundary, even 15 to 30 minutes, to ask questions live before the outgoing engineer signs off.
• Written handoff template: a standard document filled at every handoff covering open incidents, any systems in a degraded state, changes deployed in the last shift, and explicit 'known risk' or 'do not touch' notes.
• Escalation expectations: a written policy defining what counts as page-worthy versus a handoff note, who the secondary on-call is in each region, and how long the incoming engineer has to acknowledge before it auto-escalates.
  Result: even with zero live overlap between two of the three regions, the written handoff plus the short shadow window from the middle region means each incoming on-call starts already briefed, instead of reconstructing state from raw logs.
  For non-ops roles the same mechanism applies with a different artifact, for example a design or product handoff might be a written decision log plus a recorded walkthrough rather than an incident handoff, but the principle (explicit written handoff over a live conversation) is the same.

Trade-offs and pitfalls

• Repeatedly scheduling occasional syncs at painful hours burns out whichever time zone draws the short straw. Rotate it deliberately.
• Async-only breaks down for genuinely ambiguous or high-stakes decisions. Some live channel for true emergencies still has to exist.
• A handoff template that's too heavy gets skipped under time pressure. Keep it short enough to fill in within a few minutes.
• Assuming a chat message counts as a handoff is the actual failure mode this whole approach is designed to prevent. The structured artifact is the point, not the tool it's written in.

Direct answer: This function is $O(n^2)$ time and $O(1)$ additional space. The nested for loops compare every pair $(i, j)$ with $i < j$, which is the dominant cost; the total accumulator uses constant extra memory regardless of input size.

Structured elaboration

Walking through the code:

def process(arr):
    n = len(arr)
    total = 0
    for i in range(n):
        for j in range(i + 1, n):
            if arr[i] == arr[j]:
                total += 1
    return total

• The outer loop runs $n$ times.
• For each outer iteration $i$, the inner loop runs $n - 1 - i$ times (it starts at $i+1$ and goes to $n-1$).
• Total number of inner-loop executions is $\sum_{i=0}^{n-1} (n - 1 - i)$, a classic arithmetic series.
• Each inner-loop body does $O(1)$ work (a comparison and possibly an increment).
• Space: only total, i, j, n are allocated, none of which scale with input size, so space is $O(1)$ beyond the input array itself.

Worked example

For $n = 4$ (arr = [1, 2, 1, 3]), the pairs checked are $(0,1), (0,2), (0,3), (1,2), (1,3), (2,3)$ - that's $\binom{4}{2} = 6$ comparisons, matching $\frac{4 \times 3}{2} = 6$. Verified by direct execution: the function returns 1 (only the pair arr[0]==arr[2] matches), and instrumenting a counter around the inner-loop body confirms exactly 6 comparisons ran.

Since $\frac{n(n-1)}{2} = \frac{n^2 - n}{2}$, the dominant term as $n$ grows is $\frac{n^2}{2}$, so the tight bound is $\Theta(n^2)$ time.

Trade-offs & pitfalls

• A candidate might mistakenly say $O(n)$ by only looking at the outer loop without accounting for the inner loop's range shrinking - the total work still sums to a quadratic series even though each individual inner loop is shorter than $n$.
• If duplicate-detection is the real goal, this is a textbook case for the hashing-based reduction from $O(n^2)$ to $O(n)$: a single pass building a frequency map, then a second pass (or combined pass) counting matching pairs, trades $O(1)$ extra space for a much lower time bound - at the cost of $O(n)$ auxiliary space instead of $O(1)$.
• Be precise about what "dominates": the comparison arr[i] == arr[j] is $O(1)$ only if the elements are simple values (ints); if they were, say, long strings, the comparison itself could cost $O(L)$ where $L$ is string length, changing the total bound to $O(n^2 L)$.

Direct answer
A blameless postmortem is a structured written review of an incident that treats the failure as evidence of a gap in the system rather than as evidence of a person's incompetence. It assumes everyone involved acted reasonably given the information and pressure they had at the time, and it asks 'what about the system made this possible' instead of 'who made this mistake.' A good postmortem document has a small, consistent set of sections: an incident summary and severity, a timestamped timeline, quantified impact, the root cause and any contributing factors, immediate mitigations already taken, and a list of owned, dated action items.

Structured elaboration
Each section earns its place by answering a different question a reader will actually ask:

• Summary and severity. One or two sentences so a reader who will never open the full document still knows what happened and how bad it was.
• Timeline. An objective, timestamped sequence of what happened, detected, and was done. This is the shared factual spine the rest of the document hangs off; without it, discussion drifts into competing memories.
• Impact. Quantified: how many users, how much revenue, how long, which SLOs were breached. Impact is what makes prioritization of the resulting action items defensible later.
• Root cause and contributing factors. The root cause is the condition that, if changed, would have prevented the incident; contributing factors made it more likely or worse but would not alone have caused it. Separating the two stops the document from over-claiming a single tidy cause when the real story is usually several factors lining up.
• Immediate mitigation. What was done to stop the bleeding, kept separate from the long-term fix, since these often have very different owners and timelines.
• Action items with owners and dates. Concrete, individually verifiable, and never phrased as 'be more careful.' A postmortem that ends with vague advice instead of an owned commitment produces no durable change.

The wording throughout matters as much as the structure. 'The on-call engineer missed a step in the runbook' names a person; 'the runbook did not make the required step hard to skip' names a system gap that is actually fixable. This isn't softening the facts, it's redirecting the analysis toward the thing you can change.

Worked example
An API returns errors for 45 minutes after a deploy. A blame-oriented writeup might say: "the engineer pushed a bad config and didn't test it." A blameless version says: "a config change with an invalid timeout value was deployed to production without automated validation or a staged rollout; the on-call engineer restored service in 12 minutes by rolling back. Root cause: the deploy pipeline allows unvalidated config to reach 100% of traffic in one step. Contributing factor: the config schema has no automated check for out-of-range timeout values. Action items: (1) add schema validation to the deploy pipeline, owner platform-team, due in two weeks; (2) require staged rollout for config-only changes above a defined blast-radius threshold, owner SRE lead, due in one month." Same incident, same facts, but the second version is auditable, points at fixable system gaps, and produces action items an unrelated engineer could pick up and execute.

Trade-offs and pitfalls
The most common failure is stopping the investigation at 'human error' as though that were itself the root cause. If a person did something reasonable given what they knew and the system still let it cause an outage, the real root cause is upstream: missing validation, an unclear runbook, a dangerous default. A second common failure is a postmortem so long and hedged nobody reads it. Sections should be short and factual; depth belongs in linked artifacts (logs, dashboards), not in the narrative itself.

Direct answer

Feature-store health monitoring from the model-monitoring side means watching freshness, completeness, and offline/online consistency as leading indicators of model quality, with automatic remediation and graceful degradation so a feature-store problem doesn't have to become a full model outage.

Structured elaboration

1. Freshness and completeness metrics: age of the most recent update per feature (staleness), and the rate of missing/null feature values relative to historical baseline: a feature source that's stopped updating or started returning nulls at an elevated rate is a leading indicator of model degradation, often detectable BEFORE the downstream prediction-quality impact is visible.
2. Offline/online skew detection: periodically compare a feature's value as computed by the online serving path against the same feature computed by the offline/batch path for the same entity and timestamp: a growing skew indicates the two computation paths have diverged, a common and hard-to-notice source of gradually-worsening model quality.
3. Automatic remediation: fall back to a cached (slightly stale but still reasonable) value when a feature lookup fails or returns null, rather than passing null straight through to the model; for a sustained, severe outage, pause serving for the affected model entirely rather than serving predictions built on missing critical inputs, if a safe fallback isn't available.
4. Partial-outage failover design: for intermittent null joins (a feature store partially down, some lookups succeeding and some failing), implement per-request fallback logic (cached value, a documented default, or a "low confidence" flag on the resulting prediction) and validate (via a controlled comparison, not just an assumption) that the fallback behavior doesn't silently harm business metrics more than an honest, visible degradation would.
5. Safe feature-change process: introducing or changing a feature in production without breaking things needs dual-writing (compute both old and new versions in parallel before cutting over), feature flags (an instant toggle back to the old version if the new one misbehaves), shadowing (validate the new feature's values against expectations before it actually feeds a live model), and backfills (so historical data reflects the new feature consistently, not just data going forward): skipping any of these steps is how a routine feature change becomes a training-serving mismatch incident.

Worked example

Concretely: an SRE instrumenting feature-store health exposes freshness, null-rate, and cardinality per feature, with lineage hooks linking each feature version to the models consuming it: when a downstream model's error rate spikes, the SRE can immediately check whether ANY of the feature-store health signals for that model's inputs also moved in the same window, correlating feature-store health with downstream model errors directly, rather than treating the model and the feature store as separate systems investigated independently during an incident.

Trade-offs & pitfalls

The most common mistake in fallback design is validating it only in theory ("we fall back to cached values, so we're fine") without actually measuring whether the fallback's business impact is genuinely acceptable: a stale cached feature value silently used for hours during an extended outage can degrade prediction quality just as much as an honest failure would, just less visibly, since nothing LOOKS broken from a pure uptime perspective. Validating fallback behavior against real held-out comparisons (not just assuming a fallback is automatically "safe" because it's not a hard failure) is what catches this.

Direct answer

Treat this as a constrained optimization, not a wishlist: list every cost lever, estimate each one's savings and its latency/service-level-objective (SLO) risk independently, combine the savings correctly (multiplicatively, since each lever applies to whatever cost remains after the prior ones, not additively), and sequence the lowest-risk, highest-confidence levers first so you are validating architecture changes only if the safe levers don't already close the gap.

Structured elaboration

Categorize levers by risk to latency and SLOs, not just by savings size:

• Commitment-based (reserved capacity, savings plans on predictable baseline usage): near-zero runtime risk, same infrastructure, different billing.
• Right-sizing and off-peak scheduling: low risk if headroom and monitoring are retained, touches capacity, not request-path logic.
• Caching improvements: moderate risk, changes the request path and introduces a staleness trade-off, needs a pilot.
• Consolidation or replacing a managed service: highest risk, changes topology or introduces new operational surface, needs a staged rollout with a rollback path.

Execution plan: run the low-risk levers first and measure actual savings against current spend, only reach for a higher-risk lever if the low-risk set doesn't clear the target, and size that higher-risk lever to close exactly the remaining gap rather than over-applying it.

Worked example

Assume four levers, sequenced from lowest to higher risk, each estimated independently:

Combined savings are multiplicative on remaining cost, not additive, because each lever's percentage applies to whatever spend is left after the prior levers:

$\text{remaining fraction} = (1-0.15)(1-0.10)(1-0.08)(1-0.05)$

Computing stepwise: $0.85 \times 0.90 = 0.765$; $0.765 \times 0.92 = 0.7038$; $0.7038 \times 0.95 = 0.66861$.

Remaining fraction $\approx 0.6686$, so total reduction $\approx 1 - 0.6686 = 0.3314 = 33.1\%$, clearing the 30% target with roughly 3 percentage points of margin for estimation error, using only levers with low-to-moderate individual latency risk and none requiring the highest-risk consolidation lever.

If these four levers had instead totaled, say, 24%, that is the point to reach for a higher-risk lever (service consolidation or replacing a managed component), sized with the same multiplicative method to close exactly the remaining gap, and gated behind a canary rollout given its higher risk to latency and SLOs.

Trade-offs & pitfalls

• Adding percentages linearly ($15+10+8+5=38\%$) overstates the true combined savings (33.1% here) and can make a plan look like it clears the ceiling when it doesn't, always combine sequential percentage savings multiplicatively.
• Reaching for the single biggest-percentage lever first, even when it's also the highest-risk one, instead of exhausting low-risk levers first, front-loads risk unnecessarily when a safer combination might already hit the target.
• Measuring "savings" against a stale baseline instead of current spend produces accounting surprises when finance reconciles the actual bill.
• Latency and SLO risk aren't uniform across levers, track a risk budget alongside the dollar target, a plan that hits 30% savings but blows through 15% latency increase on one lever has still failed the actual constraint.

Direct answer
Precise wording matters because a reader acts on what the words literally say, not on what the writer meant; vague or ambiguous phrasing lets each reader fill in the gap with their own assumption, and those assumptions frequently disagree.

Structured elaboration

• Ambiguity creates silent forks in understanding. Two readers of the same imprecise sentence can each walk away confident they understood it, while holding two different, incompatible interpretations, and neither realizes there's a disagreement until it surfaces later, usually at a worse time.
• Vague quantifiers are a common culprit: words like "soon," "significant," or "most" mean different things to different readers and different things in different contexts.
• Precision does not mean verbosity. A precise sentence can be shorter than a vague one; "by Thursday 5pm" is both more precise and no longer than "soon."
• To enforce more precise terminology: agree on and write down a small shared glossary for terms that get used loosely (what counts as "done," what "urgent" means for this team), review drafts specifically looking for vague quantifiers and ambiguous pronouns ("it," "this") whose referent isn't obvious, and normalize asking "what do you mean by X specifically?" in review rather than letting it pass.

Worked example
Imprecise: "We'll ship the fix soon, once we've done a bit more testing."

What actually happened: one stakeholder read "soon" as "later today" and told a customer to expect it that day; the engineering team read it as "sometime this week" because "a bit more testing" meant a multi-day regression pass. The customer was told an incorrect date because two people read the same sentence and reasonably reached different conclusions.

Precise version: "We'll ship the fix by end of day Thursday, pending a two-day regression test that starts tomorrow."

Same information, but now both readers have the same understanding, and if the regression test finds something, "Thursday" is a concrete promise that either holds or needs an explicit update, rather than a vague one that quietly slips.

Trade-offs and pitfalls

• Being maximally precise about everything is exhausting and unnecessary for low-stakes communication; reserve the rigor for statements other people will act on or make commitments based on.
• Precision can be used dishonestly too, to sound more certain than you actually are; if you genuinely don't know the date, the honest and still precise move is "I don't have a firm date yet, I'll confirm by Wednesday," not a confident-sounding guess.
• Enforcing a shared glossary only works if it's actually referenced in practice, not just written once and forgotten; it needs to show up in review habits, not just documentation.

Direct answer

The randomization unit should be the largest identity that is (a) stable over the experiment window and (b) matches the unit at which you will measure and report the outcome. For a homepage layout change with user-scoped conversion metrics, that is almost always user id when you have reliable logged-in identity; fall back to device id for logged-out mobile traffic, and treat cookie and session id as fallback-only units because they leak identity across the very boundary you are trying to hold fixed. The mechanism that turns "unit" into an actual bucket assignment is deterministic hash-based bucketing, and its main operational failure mode is touching the hash inputs (the salt or key) mid-experiment. Before any of that, though, you have to define who is even eligible to be in the experiment at all.

Structured elaboration

Defining the eligible population before choosing a unit

Unit choice is a second-order question; the first-order question is which units are even eligible to enter the experiment. For a mobile-only feature (say, a redesign shipped exclusively in the mobile app to a US audience), a desktop-only visitor cannot receive the treatment no matter which arm they land in, so randomizing across your full user base and then measuring outcomes at the account level silently dilutes the experiment: ineligible units get logged into both arms with a null "effect" (they cannot experience the change either way), which pulls the estimated treatment effect toward zero and inflates the sample size needed to detect a real one. The eligible population for a mobile-only US feature is the set of units that are (a) on the mobile platform that ships the feature, (b) in the targeted market (US), and (c) past whatever version or capability gate the feature requires; everyone outside that eligible population should be excluded from the experiment entirely, not folded into control by default. This is a distinct failure mode from picking the wrong unit: a design can choose a perfectly good unit (user id) and still be broken if a third of the "users" randomized into it were structurally incapable of ever seeing the treatment, whether the unit ultimately chosen within that eligible population is user, device, or session id.

Trade-offs by candidate unit

The two axes that matter are: does this unit stay attached to one treatment condition for the life of the experiment, and does it match the level at which you will later compute the metric. Session-level randomization on a homepage layout change fails both: a returning user can see version A on Monday and version B on Wednesday, so "the effect of the layout" is not well defined for that person, and if you then report a user-level conversion rate you are averaging over users who experienced a mix of both conditions.

Target-segment and control-group selection for a personalization test

Personalization experiments add a further wrinkle on top of eligibility and unit choice: because the treatment itself varies per person (each user's personalized experience differs from every other user's), you have to be explicit about two more things: which segment of the eligible population the test targets, and what the control group actually receives. A common setup: the target segment is the subset of eligible users with enough interaction history for the personalization model to act on (say, users with a minimum number of prior sessions); users below that threshold cannot be meaningfully personalized and should either be excluded from the test or routed to a defined fallback, rather than silently folded into a "control" group that has nothing to do with the personalization decision being tested. The control group, correspondingly, should receive a clearly defined non-personalized baseline (a fixed default ranking or layout), not "whatever the legacy system happened to show," so the measured effect is attributable to personalization itself rather than to incidental differences between the two code paths. Get target-segment or control-group definition wrong (an ill-specified segment boundary, or a control group that partially overlaps with treatment logic) and the measured lift reflects a spurious selection effect rather than the personalization algorithm's real value, no matter how correctly the underlying randomization unit and hash mechanism were implemented.

How hash-based deterministic bucketing works

In practice you do not store a per-user assignment row for every experiment. Instead you compute

$\text{bucket}(u) = \text{hash}(u \,\Vert\, \text{salt}) \bmod N$

where $u$ is the chosen unit id (user id, device id, etc.), the salt is a string unique to this experiment (often the experiment name or id), and $N$ is the number of buckets (commonly 100 or 1000 for fine-grained traffic allocation). Buckets are then mapped to arms, e.g. buckets 0-49 to control and 50-99 to treatment for a 50/50 split. Because the hash is deterministic, the same unit id always lands in the same bucket for the same salt, which is what makes the assignment reproducible without a lookup table, and salting per-experiment is what makes assignment to experiment A independent of assignment to experiment B (so the same user can be validly in many concurrent, non-interacting experiments).

Operational pitfalls

• Changing the salt or hashing key mid-experiment. This is the single most common self-inflicted wound. It re-shuffles every unit into a new bucket, silently reassigning some fraction of users from control to treatment (or the reverse) partway through. The experiment now mixes users with a clean single-arm history and users who were exposed to both arms, which is exactly the session-level contamination problem from the table above, except it is invisible unless you log assignment history.
• Reusing a salt across experiments. If two unrelated experiments accidentally share a salt (or one is a substring of the identifier used in the other), their bucket assignments become correlated instead of independent, which breaks the assumption that concurrent experiments do not interfere with each other.
• Changing $N$ or the bucket-to-arm mapping. Even without touching the salt, resizing the traffic split mid-flight (e.g., ramping from 5% to 50%) moves units across the arm boundary unless the mapping is designed to be monotonic (new traffic is added to existing arms rather than everyone being rehashed).
• Identity churn. A user id that gets merged, deleted, or re-issued (account merge, logout/login cycles that mint a new anonymous id) effectively becomes a new hash input mid-experiment, which has the same effect as a salt change for that user.

A finer-grained alternative: per-impression randomization

Every unit above is a person-shaped identity. Some teams instead randomize at the impression level, assigning a fresh coin flip to each page view or ranking request rather than to a person. This is occasionally used for high-frequency, low-persistence decisions (e.g., which of several ranking variants to serve on a given request) where you explicitly do not want a stable per-user experience. It is a different trade entirely from the table above: it eliminates any notion of "this user's assigned arm" (so it cannot answer a question about a durable, user-perceived change like a homepage layout), and it introduces strong intra-user correlation in the outcome data, since one person's many impressions are not independent draws, which inflates the effective variance if you naively treat impressions as independent observations in the analysis. Per-impression randomization is the right tool only when the thing being tested is meant to vary within a single user's experience; for a homepage layout, where the goal is to measure how a stable person-level experience changes behavior, it is the wrong granularity.

Detecting and correcting unit-mismatch after the fact

1. Assignment-churn audit. From the exposure logs, compute the fraction of units that were logged under more than one arm during the experiment window. A near-zero rate is expected; anything material indicates contamination.
2. Pre-period balance check. Compare the two arms on metrics measured before the experiment started (metrics that could not possibly be affected by treatment). An imbalance signals a broken randomization, not a broken hash necessarily, but it is the same diagnostic.
3. Sample ratio mismatch check on the realized split, i.e., does the observed 50/50 (or intended ratio) actually hold at the analysis unit. A skew is a strong signal that the bucketing pipeline itself misbehaved.
4. Timeline reconstruction. If churn is found, check the deployment log for the experiment: a salt, key, or bucket-count change on a specific date will produce a visible step change in the churn-rate-by-day series.
5. Correction paths, in order of preference. Analyze by first-observed assignment only (treat each unit's initial exposure as its assignment, i.e., an intention-to-treat style rule, and accept the resulting dilution of the effect estimate); if the break has a clean date, restrict the analysis window to the stable period before or after it; if contamination is pervasive, drop the experiment's results for the affected window and rerun rather than trying to model around a broken assignment mechanism, since any post hoc adjustment for a data-dependent unit-mismatch is itself a source of bias.

Worked example

Suppose an app-only feature was randomized by session id and you are asked to sanity-check it before trusting the readout. You pull exposure logs and count, per user, the distinct arms they were logged under: 92,000 users saw only control, 91,500 saw only treatment, and 6,500 saw both. Churn rate is $6{,}500 / (92{,}000+91{,}500+6{,}500) \approx 3.4\%$. That is a directly computed, reproducible number from the logs, not an assumption, and a value that high on a homepage-layout test (where the same person plausibly returns within the experiment window) is enough on its own to recommend re-running at user-id granularity rather than trying to salvage the session-level readout.

Trade-offs and pitfalls

• Choosing the "purest" unit (household) is not free: fewer independent units means higher variance per unit, so the same absolute effect needs more households than it would need users to reach the same precision. Unit choice is a bias-versus-noise trade, not a pure bias fix.
• A cookie- or device-based fallback is a compromise you should name explicitly to stakeholders, not a silent substitute for user id; report the estimated multi-device contamination rate alongside the headline result.
• An eligible population that is defined too loosely (e.g., randomizing all traffic instead of just the mobile-only, in-market segment) produces the same kind of diluted, biased-toward-zero readout as a bad unit choice, even when the unit itself is correct.
• Do not "fix" detected contamination by re-including the mixed-exposure users with a different weighting scheme chosen after seeing which way it moves the result; decide the exclusion or ITT rule before looking at the treatment effect.

Direct answer. Resample the positive and negative examples separately, WITH replacement, at their original counts (stratified resampling), recompute AUC on each resample, and take the empirical percentiles of the resulting distribution as the confidence interval.

Code (executed and verified: the point estimate always falls inside its own CI, as it must).

import numpy as np
from sklearn.metrics import roc_auc_score

def bootstrap_auc_ci(y_true, y_score, n_boot=2000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    y_true, y_score = np.asarray(y_true), np.asarray(y_score)
    pos_idx = np.where(y_true == 1)[0]
    neg_idx = np.where(y_true == 0)[0]
    point = roc_auc_score(y_true, y_score)
    boots = np.empty(n_boot)
    for b in range(n_boot):
        idx = np.concatenate([
            rng.choice(pos_idx, size=len(pos_idx), replace=True),
            rng.choice(neg_idx, size=len(neg_idx), replace=True),
        ])
        boots[b] = roc_auc_score(y_true[idx], y_score[idx])
    lo, hi = np.percentile(boots, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return point, lo, hi

Worked example (recomputed on 400 synthetic points, 1,000 bootstrap resamples). Point estimate AUC = 0.8055, 95% CI = (0.7623, 0.8480); the point estimate falls inside its own interval, as any correctly-computed bootstrap CI must.

Structured elaboration: iterations, percentile vs bias-corrected. A few thousand resamples (1,000-10,000) is typical; below a few hundred, the percentile estimates at the tails (2.5th/97.5th) get noisy, and there are usually diminishing returns much past 10,000 unless you need a very tight tail (e.g. a 99.9% interval). The plain PERCENTILE interval used above just takes the empirical 2.5th/97.5th percentiles of the bootstrap distribution directly; a BIAS-CORRECTED (and accelerated, "BCa") interval additionally corrects for skew and for a systematic bias between the bootstrap distribution's median and the original point estimate, which matters most when the metric's sampling distribution is visibly asymmetric (common for AUC on a small or very imbalanced dataset): the percentile method can then be noticeably too narrow or shifted relative to the true interval.

Trade-offs and pitfalls: runtime and memory at scale. Each bootstrap resample recomputes AUC, which itself costs O(n log n) for the sort, so n_boot resamples cost O(n_boot × n log n) total; at very large n (tens of millions of rows), this becomes the dominant cost of the whole evaluation, and a common mitigation is to bootstrap over a fixed-size SUBSAMPLE of the full evaluation set rather than the full set itself, accepting a somewhat wider (more honest, since it reflects the subsample size) interval in exchange for tractable runtime. Memory-wise, keeping only the resampled AUC values (one float per resample) rather than materializing each resampled dataset is what keeps this from becoming a memory problem even at large n_boot.

Direct answer. Intent-revealing naming means a name tells you what a thing is or does without needing to read its implementation or a comment. daysSinceLastLogin beats d; isEligibleForDiscount beats flag. It matters more as a codebase grows because you spend far more time reading names than writing them, and a bad name actively lies to the next reader instead of just failing to help.

What makes a name intent-revealing

• It answers what, not how: activeUsers not usersWhereStatusEquals1.
• It avoids disinformation: don't call a List a userMap, don't call something temp if it's the actual result.
• It's precise about units and shape: timeoutMs not timeout; userIds (plural) for a collection.
• It's searchable: single letters and abbreviations (d, usrCnt) can't be grep'd for meaningfully across a large repo.
• It's consistent: pick one verb per concept (fetch vs get vs retrieve) and use it everywhere.

Two examples

1. Poor: def calc(u, d) where u is a user and d is a number of days. Better: def days_until_renewal(user: User) -> int. The better name tells you the return value's meaning and unit without opening the function body.
2. Poor: if (flag2) { ... } where flag2 toggles whether a discount applies. Better: if (is_first_time_customer_discount_eligible). The better name turns a branch you'd otherwise have to trace back to its assignment into something readable in place.

Why it matters more at scale

On a small script you hold the whole thing in your head, so a bad name costs you nothing. On a codebase with dozens of contributors and files you'll never open again, a name is the ONLY interface most future readers get before they decide whether they understand enough to change something safely. A clear name is effectively free documentation that can't go stale the way a comment can.

Trade-offs and pitfalls

• Longer isn't always clearer: numberOfActiveUserSessionsCurrentlyOpen is worse than activeSessionCount. Aim for precise, not verbose.
• Don't encode types in names (Hungarian notation) in a language with a type system already doing that job; it just adds noise that can drift out of sync with the actual type.
• Renaming is cheap with modern IDE tooling (safe rename across the codebase), so there's little excuse to leave a name you already know is misleading; the excuse 'it's used everywhere so I can't change it' is usually solvable with an automated rename, not a reason to give up.