Mid-Level Machine Learning Engineer Interview Preparation Guide (FAANG Standards)

Direct answer
Start with a naive baseline (predict the mean/median target), then a simple linear model, then a well-tuned gradient-boosted tree ensemble, escalating only when each step's results justify the added complexity of the next.

Structured elaboration
The naive baseline (always predict the training mean) tells you the floor: any model that can't beat it is broken, and it also gives you a reference point for how much of the target's variance is even explainable. A simple linear model next tells you how much signal is captured by an additive, mostly-linear relationship, if linear regression already gets you most of the way to a reasonable error, that's valuable information suggesting the problem may not need much complexity at all. A well-tuned gradient-boosted tree ensemble comes next as the strong general-purpose tabular choice, capturing non-linearities and interactions the linear model couldn't. Only escalate further (heavier feature engineering, ensembling multiple model families, or a neural network) if the gap between the current best model and your target/business-required error is still meaningful after the boosted-tree stage, and if that gap is worth the added complexity to close.

Worked example
Baseline MSE (predict the mean) is 500; linear regression gets to 320 (a meaningful chunk of signal is roughly linear); a tuned gradient-boosted model gets to 240 (further real gain from interactions); if the business need is an MSE under 200, that's the signal to invest in more feature engineering or a more complex ensembling approach; if 250 was already good enough, stopping at the boosted-tree stage is the right call rather than chasing further complexity for diminishing returns.

Trade-offs & pitfalls
Skipping the baseline/linear stages to jump straight to the most sophisticated model available is a common instinct but it costs you the diagnostic information each earlier stage would have given you about how much of the problem is genuinely non-linear versus how much complexity is just adding noise-chasing capacity.

Direct answer

Before your work depends on it, put a written interface contract in place (the shape of the data or API, error cases, and versioning), a single named owner on each side, and an SLA (service level agreement: the vendor's contractual uptime/response commitment) for questions and changes with a defined escalation path. Then build against a mock or stub (a fake stand-in for the real API that returns data matching the agreed contract, so your team can build and test without waiting on the real thing) that matches that contract, so a late dependency delays true integration, but doesn't block your team's progress.

Framework

Before you start building. Agree the contract explicitly (schema, error handling, versioning), name one owner per side rather than 'the team', and set an SLA for response time and change turnaround, with an escalation path if it slips.

While you wait. Build and test against a mock or stub that matches the agreed contract, so your team keeps moving. Pair it with automated contract tests, so if the mock and the real dependency drift apart, you find out at build time instead of at release.

Internal-team dependency vs external vendor dependency. The mechanics differ once the other side is a vendor rather than a team you can walk over to.

Worked example

Situation: a product depends on a vendor-managed API (for example a payments or identity provider). The vendor's contract commits to 99.5% availability, but the product's own reliability target requires 99.95%.
Quantifying the gap: a year has 8,760 hours. At 99.5% availability, permitted downtime is 0.5% of 8,760 = 43.8 hours per year. At 99.95%, permitted downtime is 0.05% of 8,760 = 4.38 hours per year. The vendor's contract therefore permits about 43.8 minus 4.38 = 39.42 hours per year more downtime than the product can actually tolerate.
Action: negotiated for a higher committed SLA where possible; where the vendor would not move the number, negotiated a compensation or credit clause tied to a downtime threshold, documented in writing. Regardless of the contract terms, added caching on the read path so a short vendor blip doesn't cascade immediately, and a fallback path that degrades the feature gracefully instead of erroring during an outage window.
Result: the contract negotiation raises the ceiling on paper, but the caching and fallback layer is what actually protects users during the gap between what the vendor promises and what the product needs, since a credit clause compensates you after an outage, it doesn't prevent one.

Trade-offs and pitfalls

• Mocks and stubs only help if kept in sync with the real contract. A stale mock creates a different kind of surprise at integration time.
• Vendor SLA credits are usually a small fraction of the real cost of downtime (lost trust, lost usage). Treat them as compensation, not as risk mitigation on their own, and pair them with technical fallbacks.
• Applying heavy contract-and-SLA process to a short, low-risk internal dependency slows down partners who need speed more than ceremony. Calibrate the rigor to the risk and duration of the dependency, not the same weight for every one.

Direct answer: When training under severe class imbalance with very few positive examples, feature selection needs extra care against overfitting the selection itself to the small number of positive cases, favoring methods and workflows that check stability across resamples and external validation over a single fit's ranking.

Structured elaboration:
With very few positives, filter methods (mutual information, correlation) computed on the full imbalanced dataset can be noisy specifically because the positive class's statistics are estimated from so few examples; wrapper methods (RFE) similarly risk fitting to idiosyncrasies of the small positive set rather than a genuine pattern; embedded methods share the same underlying data-scarcity risk. A practical workflow: use resampling (oversampling the minority class, or a resampling-aware cross-validation scheme) specifically for the SELECTION step, not just the final model fit; run stability checks (does the selected feature set stay consistent across several different resamples, rather than trusting one run); and validate on an external, held-out set that wasn't used for selection at all, specifically checking that the selected features generalize to genuinely new rare-positive examples, not just the ones the selection process already saw.

Worked example: With only a few dozen positive examples in a large dataset, a single filter-method run might rank a feature highly purely because of how a handful of positive examples happen to look, a ranking that could shift substantially if even one or two of those rare positive examples were different; running the same filter across several resampled subsets and keeping only features that consistently rank highly across most of them is a direct, practical defense against over-trusting a ranking built from so little positive-class data.

Trade-offs and pitfalls: The temptation to trust a single selection run's output is stronger under severe imbalance specifically because there's SO little positive data that any single run feels like it's using "all the signal available"; the discipline of stability-checking across resamples matters MORE here, not less, precisely because the underlying data is so sparse for the class that matters most.

Direct answer. An intermittent multi-GPU failure (one node's CUDA OOM, or results diverging across otherwise-identical nodes) needs evidence gathered from every rank, not just the one that visibly failed, because the actual fault often originates on a DIFFERENT rank than the one that shows the symptom.

Debugging plan.

1. Collect NCCL, CUDA, and system logs from every rank, not just the failing one, and turn them on BEFORE the run, since none of this can be recovered retroactively. A rank that silently produces a NaN, for instance, often only shows up as an OOM or a hang on a DIFFERENT rank once the collective operation (all-reduce) tries to synchronize with it. The three layers each need their own switch.
   • NCCL / collectives: NCCL_DEBUG=INFO (plus NCCL_DEBUG_SUBSYS=ALL when you need the ring topology) shows which ranks joined which communicator and where a collective stalled. Pair it with TORCH_NCCL_ASYNC_ERROR_HANDLING=1 so a rank that dies tears the job down with an error instead of leaving every other rank blocked forever in the all-reduce, and TORCH_DISTRIBUTED_DEBUG=DETAIL to get shape and dtype mismatches across ranks reported as errors rather than as silent corruption.
   • CUDA: on a REPRO run (not the production one, because it serializes kernel launches and changes timing), set CUDA_LAUNCH_BLOCKING=1 so the traceback points at the kernel that actually failed rather than at whatever later call happened to synchronize. For the OOM specifically, wrap the step in a handler that dumps torch.cuda.memory_summary() at the moment of failure, and for a repeat offender turn on the allocator trace with torch.cuda.memory._record_memory_history() and dump it with torch.cuda.memory._dump_snapshot(), which gives you the call sites holding every live block instead of a single total.
   • System: check the kernel log on the suspect node (dmesg -T) for Xid errors and for the host OOM killer, and nvidia-smi -q -d ECC,TEMPERATURE,POWER for uncorrectable ECC counts, retired pages and thermal or power throttling. This layer is the one people skip and it is the one that answers the question actually being asked, which is why THIS node. A repeating Xid or a rising uncorrectable-ECC count on exactly one node turns a week of software debugging into a node replacement, and a host-side OOM kill explains a node dying with no GPU memory problem at all.
2. Separate the three memory failure shapes: a genuine leak, fragmentation, and a legitimate peak-allocation spike. They look identical at the moment of the OOM and completely different across many steps, so log TWO numbers per step, not one: memory ALLOCATED (the bytes currently held by live tensors) and memory RESERVED (the bytes the caching allocator is holding from the driver, including free-but-cached blocks). In PyTorch those are torch.cuda.memory_allocated() and torch.cuda.memory_reserved(). On a 16 GiB device the three shapes read like this:

step              1      50     100    150    200
LEAK        alloc 8.1    8.4    8.7    9.0    9.3   (GiB, monotonic climb)
            resv  8.6    8.9    9.2    9.5    9.8
SPIKE       alloc 6.2    9.8    6.2    9.8    6.2   (sawtooth, returns to baseline)
            resv  9.9    9.9    9.9    9.9    9.9
FRAGMENT    alloc 6.0    6.0    6.1    6.0    6.0   (flat, far below capacity)
            resv  9.5   10.2   10.9   11.4   11.8   (climbs away from alloc)

• Leak: allocated climbs monotonically across steps and never returns to baseline. Something is retaining references (a loss tensor accumulated into a Python list without .detach() or .item(), a growing cache, a hook holding activations). Fix the retention. Raising the memory ceiling only buys steps.
• Spike: allocated is a sawtooth that returns to the same baseline every step, with the peak driven by a particularly large batch or an activation-checkpointing boundary. Reserved sits flat at the high-water mark. This is normal behavior against an unlucky ceiling, and reducing batch size or enabling activation checkpointing genuinely fixes it.
• Fragmentation: allocated stays flat and well below device capacity while reserved climbs away from it, and the OOM message itself gives it away, reading something like "tried to allocate 2.00 GiB, 3.50 GiB free": there IS enough free memory in total, just not in one contiguous block. The other tell is that failure depends on allocation ORDER rather than step count, so it can fire at step 12 on one run and step 400 on the next, and it is strongly associated with varying tensor shapes (variable sequence lengths, ragged batches) that make each allocation a slightly different size. The fixes are different in kind from the other two: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (or tune max_split_size_mb) so the allocator stops carving unusable slivers, call torch.cuda.empty_cache() at a safe step boundary to hand cached blocks back to the driver, bucket or pad variable-length inputs so allocation sizes repeat instead of drifting, and pre-allocate the large buffers once at start-up. Note what is NOT on that list: reducing batch size, which is the standard reflex and does not reliably help here, because the problem was never total capacity.

Confusing any two of the three leads to the wrong fix, and the fragmentation case is the one where the wrong fix is most tempting, since the allocator is reporting plenty of free memory while the allocation still fails.
3. Validate that batch size and sharding are actually consistent across ranks. A configuration bug (one node launched with a stale config, or an uneven data-sharding split) can silently give one rank a larger effective batch than the others; log the actual batch size and shard boundaries each rank believes it has, and diff them across ranks rather than assuming the launch config was applied uniformly everywhere.
4. Isolate WHICH rank produces an anomaly (for example a NaN) in a multi-GPU job: add a lightweight per-rank check right after the forward pass (before the collective all-reduce) that logs a boolean "this rank saw a NaN" flag, tagged with the rank ID, and aggregate these flags centrally. This turns "the job produced a NaN somewhere" into "rank 3 produced the NaN, ranks 0/1/2 were clean," which narrows the investigation from the whole cluster to one node's data shard, environment, or hardware.

Short-term mitigation to keep the job running while investigating. Reduce the per-rank batch size (or enable gradient accumulation to compensate) to lower peak memory pressure and buy headroom while you investigate the OOM's root cause, and enable periodic checkpointing (if not already in place) so an eventual failure doesn't lose the whole run's progress. This is explicitly a stopgap, not a fix: if the true cause is a genuine leak or a sharding bug, reducing batch size only delays the eventual failure, and if the cause is fragmentation it may not delay it at all, since a smaller batch changes allocation sizes without making the free memory any more contiguous. In that case the cheapest stopgap is instead the allocator setting plus a periodic empty_cache() at a step boundary.

Direct answer

For a daily batch scoring job to run reliably without silently redoing or skipping work, three things must all hold: every run is idempotent (re-running a day overwrites cleanly instead of duplicating or drifting), every run reads a fixed, known snapshot of its inputs rather than "whatever happens to be in the table right now," and a partially-completed run is never indistinguishable from a fully-completed one to anything downstream.

Structured elaboration

Idempotent writes. Write scored rows to a staging area keyed by a run id, then merge or upsert into the production table keyed by something like (user id, score date). Re-running the same day overwrites the same keys instead of appending duplicates.

A fixed input snapshot. Pin the exact feature snapshot (a partition, or an as-of timestamp) that the job reads. A retry must read the identical input the original attempt read, not a feature table that has changed underneath it since. Without this, "the same run" run twice can legitimately produce two different, both-valid-looking results, which is a much harder failure to catch than an obvious crash.

Run bookkeeping. A run-metadata table recording run id, status, which input snapshot was used, and expected-versus-actual row counts. This is what lets a retry answer "did the previous attempt actually finish," instead of inferring it from whether the output table happens to have rows in it.

An explicit completion signal. Downstream consumers should key off a completion pointer or manifest that only flips after validation passes (row count matches the expected cohort size, a basic distribution sanity check), not off "the table exists for today's date." A job that crashes at 60% through can leave a partially-populated table that looks superficially fine to anything checking only for existence.

Failure handling. On failure, the job should fail LOUDLY (an alert, not a silent retry-forever loop) and the retry must be safe to run to completion using the same run id and the same input snapshot. A scheduler that treats "the process exited with code 0" as success, without checking the completion manifest, is exactly how a partial run gets silently treated as complete.

Worked example

Walk through one day's run. The job is scheduled at 02:00 against the feature snapshot as of 01:00, covering 50 million users. At 60% through, it crashes on a transient timeout, having written 30 million rows to staging. The orchestrator retries using the SAME run id and the SAME 01:00 snapshot; the retry reprocesses all 50 million users (an idempotent overwrite of staging, not an append), completes, and the validation step checks the written row count against the expected cohort size before anything downstream is allowed to see it. Only after that check passes does the atomic merge into production happen and the completion manifest flip; downstream systems only ever observe either "yesterday's complete data" or "today's complete data," never the transient 30-million-row partial state.

For sizing the compute layer to actually finish inside a batch window, reason from throughput, not a wall-clock guess. Assume (an illustrative planning estimate, not a measured figure) a single worker can score 2,000 rows per second, and the batch window is 4 hours (14,400 seconds):

So at least 2 workers running in parallel are the bare minimum to fit inside the window; in practice you'd provision well above that (for example 8 to 16 shards) so a retry after a partial failure still has enough headroom to complete inside the same window instead of blowing the schedule.

Trade-offs & pitfalls

A full-table merge/upsert can get expensive at large scale; if the whole cohort is recomputed every day anyway, an atomic partition-level swap is often cheaper than a row-level upsert. Pinning a fixed input snapshot trades some freshness for reproducibility, a real cost if the underlying feature store updates intraday, so the staleness bound this introduces should be documented, not assumed away. The most common wrong turn is relying on "the table has today's date populated" as the signal that a run is complete, instead of an explicit row-count and validation gate; that pattern is exactly how a partial run silently passes as done.

Direct answer
At high incident volume, right-sizing postmortem effort means reviewing incidents proportionally to their severity and learning value rather than giving every incident the same heavyweight treatment, since a full deep-dive on every minor blip both burns out reviewers and dilutes attention from the incidents that actually deserve it.

Structured elaboration

• Tier the review depth by severity and novelty. High-severity or novel-pattern incidents get the full treatment: timeline reconstruction, root cause and contributing factors, cross-team facilitation. Low-severity, well-understood, or clearly one-off incidents get a much lighter 'mini' review: a short written summary with a root cause and, if warranted, one action item, no meeting required.
• Rotate reviewers rather than relying on the same few people. Concentrating review responsibility on a small group both burns them out and creates a bottleneck; distributing it (with a shared template and light training) keeps quality consistent while reducing individual load.
• Automate triage where the pattern is well understood. If a category of incident has occurred many times with the same known cause, an automated or templated mini-postmortem that flags it as a known, tracked pattern (rather than requiring fresh analysis every time) frees up reviewer time for genuinely novel incidents.
• Track a pattern-level view, not just per-incident. A large volume of small, similar incidents is itself a signal worth its own dedicated (heavier) review, even if none of them individually crossed the severity threshold for a full postmortem, since the aggregate pattern is often more informative than any single instance.
• Measure whether this is actually preserving learning value, not just reducing workload. Track whether recurrence rates for previously-reviewed incident classes stay flat or improve even as review depth for minor incidents drops, to confirm the lighter-touch approach isn't quietly letting real risk go unaddressed.

Worked example
An organization runs roughly 2,000 incidents a month and full postmortems have become a rubber-stamp exercise nobody has time to do well. The fix: define three tiers. Tier 1 (high severity or genuinely novel pattern, maybe 5% of incidents) gets a full facilitated postmortem within a defined turnaround. Tier 2 (moderate severity, somewhat familiar pattern, maybe 25%) gets a lightweight async writeup by the on-call responder, reviewed by a rotating peer within a week, no live meeting required unless something surprising surfaces. Tier 3 (low severity, well-understood and recurring pattern, the remaining ~70%) gets an automated, templated log entry tagging the known category, with no individual analysis required unless the volume of that specific category spikes, which triggers escalation to a full pattern-level review. Reviewer rotation is enforced across teams so no single person is doing more than a defined share of Tier 1 and Tier 2 reviews in a given month.

Trade-offs and pitfalls
The biggest risk of this approach is under-reviewing something that seemed minor in isolation but was actually an early instance of a bigger, developing problem; the pattern-level tracking (watching for a spike in a normally-quiet Tier 3 category) is what catches that, and skipping it is the most common mistake when teams implement tiering purely to save time.

Direct answer: BFS visits nodes level-by-level outward from the source (using a queue), while DFS plunges as deep as possible along one path before backtracking (using a stack, explicit or via recursion). Both are $O(V + E)$ time on an adjacency list; on an adjacency matrix both become $O(V^2)$ because checking each vertex's neighbors costs $O(V)$ regardless of actual edge count. Space differs: BFS's queue can hold up to $O(V)$ nodes at the widest level, while DFS's stack depth is at most $O(V)$ in the worst case (a long path) but often much less for a bushy, shallow graph.

Structured elaboration

• BFS: typically implemented with a queue and a visited-set; explores all neighbors of the current frontier before moving to the next level. Naturally finds the SHORTEST path (by edge count) in an unweighted graph, because it discovers nodes in strictly increasing distance order from the source.
• DFS: typically implemented with an explicit stack or recursion (which uses the call stack implicitly); explores one branch fully before backtracking. Naturally suited to problems about structure/connectivity - detecting cycles, topological sorting, finding connected components - because it naturally tracks the "path so far."
• On an adjacency list, both visit each vertex once ($O(V)$) and traverse each edge once (or twice for undirected graphs) ($O(E)$), giving $O(V+E)$.
• On an adjacency matrix, finding a vertex's neighbors means scanning its entire row, $O(V)$ per vertex regardless of how many edges actually exist, giving $O(V^2)$ total - the representation choice materially changes the complexity class for sparse graphs.

Worked example

For a sparse graph like a social-network friend graph (say $V = 10^6$ users, average degree $d = 100$, so $E \approx 5 \times 10^7$ for an undirected graph): adjacency-list BFS/DFS costs $O(V+E) \approx 5.1 \times 10^7$ operations. The same traversal on an adjacency matrix would cost $O(V^2) = 10^{12}$ operations - a roughly 20,000x blowup, entirely from the representation choice, not the algorithm. This is why adjacency lists are the default for real-world sparse graphs, and adjacency matrices are reserved for dense graphs or when O(1) edge-existence lookup is specifically needed.

Trade-offs & pitfalls

• BFS is the right choice for shortest-path-in-unweighted-graph and "closest/nearest" queries; DFS is the right choice for exhaustive exploration, cycle detection, and topological ordering.
• DFS's recursive implementation risks stack overflow on deep graphs (a long chain); an explicit-stack iterative version avoids this at the cost of slightly more code.
• Don't default to adjacency matrix out of habit - for any graph where $E \ll V^2$ (the common case), it wastes both time and $O(V^2)$ memory versus an adjacency list's $O(V+E)$.

Direct answer
Training on 1%-positive data means combining sampling, loss reweighting, and evaluation-metric choices together, since no single lever fixes severe imbalance alone, and every choice trades bias against variance or against probability calibration in a specific, predictable way.

Structured elaboration
Sampling: stratified mini-batches (guaranteeing a minimum number of positives per batch) stabilize the gradient signal batch to batch; oversampling positives increases their effective representation but risks overfitting to duplicated examples; undersampling negatives reduces majority-class dominance but discards real negative-class diversity, increasing variance; a moderate hybrid of the two is a common practical compromise.
Loss modifications: class-weighted cross-entropy (inverse-frequency weighting) or focal loss both up-weight the rare positive class's contribution to the gradient; focal loss specifically ALSO down-weights easy, already-confident negatives regardless of class, which helps when the overwhelming negative majority would otherwise dominate the gradient purely through sheer volume even without a class-frequency imbalance argument.
Architecture: a two-stage cascade (a high-recall, cheap first-stage filter followed by a more precise second-stage classifier) can be a practical way to handle SEVERE imbalance operationally, effectively re-balancing the class ratio the expensive second stage actually has to handle; ensembling reduces variance from any one model's overfitting to the specific (oversampled or undersampled) training distribution.
Evaluation metrics you would actually TRUST: precision-recall curves and area-under-PR (AUPRC), not ROC-AUC, since ROC-AUC can look deceptively strong under severe imbalance (a large true-negative pool inflates the false-positive-rate axis's apparent stability); calibration metrics (Brier score, reliability diagrams) specifically because class-reweighting and focal loss both distort the model's OUTPUT PROBABILITIES away from true likelihoods, even when they improve ranking-based metrics.

Worked example
A concrete illustration of the bias/variance/calibration trade-off: oversampling the positive class 20x to roughly balance a 1%-positive dataset will typically REDUCE bias toward the majority class (the model now sees positives often enough to learn their pattern) but INCREASES variance, since the model has, in effect, seen far FEWER distinct positive examples than the oversampled count suggests, risking memorization of the specific duplicated positives rather than the underlying pattern; and it will also DISTORT calibration, since the model's predicted probabilities now reflect the ARTIFICIAL 50/50-ish training distribution rather than the true 1% prevalence, requiring a POST-HOC calibration step (Platt scaling or isotonic regression, fit on a held-out set with the TRUE prevalence preserved) before the raw output probabilities can be trusted for any downstream decision that depends on them being genuine probabilities.

Trade-offs & pitfalls
A common mistake is tuning and reporting final metrics on a resampled (balanced) validation set that no longer reflects the true 1% prevalence; the model's reported precision and recall will look systematically different (often much better) on a balanced validation set than they would on data matching the REAL deployment prevalence, so a dedicated validation set preserving the true class ratio is essential for any number that will actually inform a production decision. A second common gap is choosing a decision threshold using accuracy or a default 0.5 cutoff; under 1% positive prevalence, the operating threshold needs to be chosen deliberately against the actual cost trade-off between false positives and false negatives, using the precision-recall curve, not left at a default that made sense for a roughly-balanced problem.

Direct answer

A cascade, a cheap filter first and an expensive model only on the fraction it is unsure about, wins whenever the input mix is skewed toward easy cases, because it lets you pay the expensive model's cost only on the hard tail. A single large model wins when you cannot build a cheap filter with high enough recall on the escalation decision itself, since a weak filter silently drops cases that needed the expensive model's judgment.

Structured elaboration

• Cascade design: a cheap stage-1 classifier resolves confident cases directly; anything below a confidence threshold is escalated to the expensive stage-2 model.
• What matters most: not the cascade's average accuracy, but the stage-1 filter's recall on "this needs escalation," a miss there means the expensive model never sees a case that needed it.
• Complexity cost: two models to version and monitor for drift, and request latency now varies by path length, some requests take one hop, some take two.
• Single model: simpler to operate, but every request pays the expensive model's full cost, even the easy 90%.

Worked example

10 million posts/day, cheap filter $0.0001/post, expensive model $0.01/post, and the filter confidently resolves 90% while escalating the other 10%:

$\text{cascade cost} = 10{,}000{,}000 \times \$0.0001 + 1{,}000{,}000 \times \$0.01 = \$1{,}000 + \$10{,}000 = \$11{,}000/\text{day}$
$\text{all-expensive cost} = 10{,}000{,}000 \times \$0.01 = \$100{,}000/\text{day}$

That is roughly 89% cheaper. But if the filter's recall on "needs escalation" is only 95% against a true escalation population of 1,000,000:

$\text{missed escalations} = 1{,}000{,}000 \times (1 - 0.95) = 50{,}000/\text{day}$

50,000 posts a day get moderated by the wrong tier, a real cost that has to be weighed against the $89,000/day saved.

Trade-offs and pitfalls

Cascades only make sense if you can actually measure the filter's recall on the escalation decision, not just its overall resolve rate. Latency variance (a two-hop tail versus a one-hop median) can also blow past a p99 budget if not accounted for.

What the interviewer probes next

Expect follow-ups on how you would set and validate the confidence threshold, how to monitor drift when the two models retrain on different schedules, and how adversarial inputs crafted to slip past the cheap filter change the design.

Splitting strategy (before any resampling or reweighting): with well under 1% positive rate, a plain random train/test split risks landing a fold with zero or near-zero positive examples, which silently breaks both training (the model never sees enough positive signal in some folds) and evaluation (a fold with 2 positives out of 50,000 makes any per-fold metric wildly noisy). Always use a STRATIFIED split (stratified train/val/test, and stratified k-fold for cross-validation) so every split preserves close to the true positive rate, and explicitly check that no fold ends up with zero positives before trusting its metrics; with severe enough imbalance (say, under 0.1%), consider more folds (e.g., 10-fold instead of 5-fold) specifically so each fold still contains a usable number of positive examples.

Briefly, common methods to handle class imbalance:

• Random oversampling: duplicate minority examples until balance.

  • Pros: simple, preserves all original data.
  • Cons: overfitting risk, larger dataset → longer training, storage/IO costs.

• Random undersampling: drop majority examples.

  • Pros: fast, smaller dataset, less training time.
  • Cons: loses information, can underfit, unstable if dataset small.

• SMOTE (Synthetic Minority Over-sampling Technique): synthesize new minority samples by interpolating neighbors.

  • Pros: less naive than duplication, reduces overfitting to exact examples.
  • Cons: can create unrealistic samples for complex data (e.g., images/text), may blur class boundaries.

• Class weights: give higher loss weight to minority classes during training.

  • Pros: no data duplication, minimal IO/memory cost, works well with deep nets and large datasets, integrates cleanly with training frameworks.
  • Cons: may need tuning, can cause training instability if weights too large.

• Focal loss: down-weights easy examples, focuses learning on hard/rare examples.

  • Pros: effective when many easy negatives overwhelm loss (e.g., dense detectors).
  • Cons: adds hyperparameters (gamma, alpha), not universally better than simple weighting.

• Threshold-moving (decision threshold adjustment): change prediction threshold per class after training to meet precision/recall trade-offs.

  • Pros: simple, no retraining, good for calibrating operating point.
  • Cons: doesn’t change learned representation; limited if model never learned minority features.

Practical trade-offs summary:

• For small/moderate tabular datasets, SMOTE + careful validation can help.
• For large-scale deep-learning on images/text, class weights or focal loss are preferable to oversampling because they avoid multiplying data, preserve training throughput, and let the model learn from full diversity of majority class.
• Undersampling is acceptable when majority class is huge and redundant.

Which metrics to actually trust under this level of imbalance: accuracy is close to meaningless here, since a model that predicts 'negative' for every single example clears 99%+ accuracy while catching zero true positives. ROC-AUC is also misleading on its own: it can look strong (0.85+) even while precision at the threshold you'd actually operate at in production is unusably low, because ROC-AUC averages performance across ALL thresholds, most of which nobody would ever deploy at under severe imbalance. The metrics worth trusting here are PR-AUC (precision-recall AUC, which does not get inflated by the huge pool of easy true negatives the way ROC-AUC can) and precision-at-a-fixed-recall (or recall-at-a-fixed-precision), read directly off the precision-recall curve at the specific operating point you intend to run in production, not averaged across the whole curve.

When class weighting is preferable to oversampling in production:

• High-throughput training pipelines on GPUs where IO/augmentation cost matters: weighting avoids enlarging dataset and keeps batch composition natural.
• When data augmentation already exists (images/text) and synthetic duplication would produce correlated batches increasing overfitting.
• When you need reproducible, stable deployment pipelines (no extra synthetic-data generation or storage).
• When minority class examples are noisy: oversampling amplifies noise; weighting preserves original distribution while guiding loss.

In practice I start with class weights (or focal loss for extreme background/foreground imbalance), validate with threshold-moving, and only use oversampling/SMOTE if validation shows the model isn’t seeing enough minority variation.

Using the precision-recall curve to pick an operating threshold under a real business constraint (worked example): suppose the model scores 100,000 items/day, the true prevalence is 0.8% (800 true positives/day), and the manual-review team can handle 500 flagged cases per day. Walk the precision-recall curve to find the threshold whose resulting flagged volume is about 500/day (roughly the threshold above which the model predicts positive for about 0.5% of daily volume), and read off precision and recall AT THAT SPECIFIC THRESHOLD, say precision=70% and recall=44%: at that operating point, the team reviews 500 cases/day, of which 0.70*500=350 are true positives (matching the reported precision), which is 350/800=43.75% (about 44%) of the day's 800 true positives (matching the reported recall). If leadership says 44% recall is too low, the concrete lever is capacity, not the model: either grow review capacity (moving further down the PR curve toward a lower threshold trades lower precision and more flagged volume for higher recall) or keep capacity fixed and prioritize which 500 get reviewed by expected severity rather than by score alone. This is the mechanism by which 'manual-review capacity' and 'the PR curve' connect to an actual threshold decision, instead of being named separately with no bridge between them.