Anomaly and Fraud Detection Questions

Direct answer
Positive-Unlabeled learning fits a fraud dataset where you have some confirmed fraud cases (the positives) but the rest of the data is simply unlabeled rather than confirmed legitimate, since many transactions that look "not fraud" are really just never investigated, not verified clean. When some of those unlabeled negatives are actually undiscovered fraud, that label noise means a model validated against them as if they were clean will systematically underestimate both false positives (some "false positives" are actually correct catches of undiscovered fraud) and true detection performance.

Structured elaboration

• Why plain supervised learning breaks down here: standard supervised learning assumes your negative labels are correct. If a meaningful fraction of "not fraud" labels are really "never investigated, so unknown," the model is effectively being told some real fraud is legitimate, which both corrupts training and, more subtly, corrupts your evaluation metrics, since a case the model correctly flags as fraud but which was never confirmed will be scored as a false positive.
• One algorithmic approach: treat the unlabeled set as a mixture of hidden positives and true negatives, and use a method like PU bagging (training many classifiers, each treating a random subsample of the unlabeled set as negative, then averaging their scores) or a bias-corrected classifier that explicitly estimates the unknown positive fraction in the unlabeled set rather than assuming it's zero.
• How label NOISE specifically changes validation: if you validate the usual way (treating every unlabeled case as a confirmed negative), a model that's actually performing well will look artificially worse, since some of its "false positives" are real fraud your labels just haven't caught up to yet. A more honest validation approach is to specifically re-review a sample of the model's flagged-but-officially-unlabeled cases through manual investigation, rather than trusting the existing label set as ground truth for evaluation.

Trade-offs and pitfalls
PU-learning techniques generally require an estimate of what fraction of the unlabeled data is truly positive, and that estimate itself is often uncertain; get it meaningfully wrong and the bias-correction can do more harm than good. It's also worth periodically re-auditing older "unlabeled" cases as more time passes and more investigations complete, since the fraction of true positives hiding among them tends to shrink (though never fully disappear) the longer a case has had a chance to be reported.

Direct answer
Keep the fast tier lightweight and fed only by features that are already available with zero extra latency (precomputed aggregates, simple rules), have it make a coarse immediate decision (allow, block, or "send to the slow tier for a fuller look"), and let the slow tier run a heavier model asynchronously to catch what the fast tier couldn't decide confidently; keep the two tiers consistent by making the fast tier conservative (biased toward "send to slow tier" rather than a confident wrong call) and by versioning both tiers together so an update never leaves one tier reasoning about features the other tier no longer produces.

Structured elaboration

• Fast tier design: uses only features that are already computed and cached (no live lookups that could add latency), and its output is one of three outcomes: allow, hard-block, or "uncertain, defer to slow tier." Keeping "uncertain" as a first-class output, rather than forcing a binary decision, is what lets the fast tier stay fast without pretending to know more than it does.
• Slow tier design: runs a heavier model with a fuller feature set (including features that take longer to compute or require an external lookup), reviewing transactions the fast tier deferred, either before the transaction fully settles (if your business tolerates a short hold) or shortly after, triggering a compensating action (a hold, a review flag, a chargeback pre-emption) if it disagrees with the fast tier's initial allow.
• Handling a feature that hasn't finished computing in time for the fast tier: treat a missing or stale feature as a signal in itself, not silently as zero or as "average," and route transactions with missing critical features to the "uncertain, defer" outcome by default, rather than letting the fast tier make a confident decision on incomplete information.
• Keeping the two tiers consistent: version the fast and slow tier together as a single release unit, so a fast-tier feature change is never deployed without confirming the slow tier's fuller feature set (and its own model) has been validated against the same underlying feature definitions; a common integration bug is the two tiers silently drifting to use different versions of what should be "the same" feature.

Architecture sketch

flowchart TD
    A[Incoming transaction] --> B[Fast tier: lightweight model, cached features only]
    B -->|confident allow| C[Process transaction]
    B -->|confident block| D[Reject transaction]
    B -->|uncertain| E[Slow tier: heavier model, fuller feature set]
    E -->|confirms allow| C
    E -->|confirms block| F[Compensating action: hold or reverse]
    E -->|escalate| G[Analyst review queue]

Trade-offs and pitfalls
This pattern trades some latency for the slow tier's transactions (since a "deferred" case doesn't get its final answer immediately) against materially better accuracy on the harder cases; the business needs to be comfortable with the specific latency/consistency behavior for a deferred transaction, whether that means a brief hold, a provisional approval with a possible later reversal, or an immediate approval that can only trigger a downstream action (like a targeted review) rather than an outright reversal. Rolling out an update to either tier without a clear plan for the other risks a period where the fast tier's "uncertain" rate spikes or drops unexpectedly, which is worth alerting on directly.

Direct answer
Design a randomized experiment that compares the model with and without the new feature on the SAME population, using a metric that's insensitive to the fraud label's delay (for example, agreement with eventual confirmed outcomes measured only once enough time has passed for both arms' cases to be equally confirmed), rather than comparing performance measured at two different points in time, which would confound the feature's effect with any other change that happened between those two points.

Structured elaboration

• Experimental setup: run the model with the new feature and the model without it side by side on the SAME live traffic (for example, computing both scores for every transaction but only acting on one, chosen by randomization, or running both as shadow scores and comparing their agreement with eventual outcomes), rather than a before/after comparison, which would be confounded by any other change (a rule update, a shift in fraud patterns) happening in between the two periods.
• Sampling: since the new feature's real value shows up specifically on the subset of cases where it changes the decision, oversample or specifically examine the disagreement set (cases where the two model versions produce meaningfully different scores) for manual review, rather than relying purely on an aggregate metric across all traffic, most of which the new feature won't meaningfully affect either way.
• Metric choice: track a metric tied to eventual CONFIRMED outcomes (recall and precision measured only after the label-delay window has passed) rather than an immediate proxy metric, since an immediate proxy could show a spurious improvement or degradation that doesn't hold up once real confirmations arrive.
• Keeping statistical validity under label delay: define the experiment's evaluation window explicitly to end only once you'd expect confirmations for that period to be largely complete (see the separate question on label delay for how to estimate that window), and pre-register a fixed sample size or duration for the comparison, rather than "peeking" at partial results and extending or cutting the experiment based on what you're seeing, which inflates the risk of a false positive conclusion about the feature's value.

Trade-offs and pitfalls
The most common way this kind of experiment goes wrong is checking results too early, before the label-delay window has meaningfully closed, and concluding a lift or no-lift that partial, still-incomplete confirmations don't actually support; write down your evaluation window and stopping rule before looking at any results, exactly as you would for any other statistically-rigorous experiment. Also weigh the ENGINEERING cost side explicitly, a feature that adds meaningful serving latency or pipeline complexity needs a correspondingly larger detection lift to be worth shipping, not just any statistically-detectable improvement.

Direct answer
Use SHAP (SHapley Additive exPlanations) or a similar feature-attribution method to break down a specific flagged transaction's score into how much each feature contributed, which gives a fraud analyst a concrete, case-specific reason to trust or dismiss a flag, and gives a compliance stakeholder an aggregate view of which features drive the model's decisions overall, for auditing whether the model is relying on sound, defensible signals. The same technique can also catch a model that's leaning on a feature it shouldn't be, since a feature whose attribution shows outsized influence in ways that don't make domain sense is a concrete, investigable lead toward a leakage or reliability problem.

Structured elaboration

• For a fraud analyst reviewing one case: SHAP decomposes a single prediction into a per-feature contribution, showing an analyst exactly which signals pushed this specific transaction's score up (an unusual device, a velocity spike, an amount far from the account's history) rather than making them trust an opaque number, which materially speeds up manual review and builds justified trust in the model's flags over time.
• For a compliance stakeholder auditing the model overall: aggregate SHAP values across many predictions to show the model's GLOBAL feature-importance ranking, which lets a compliance reviewer confirm the model is making decisions based on legitimate, explainable signals (transaction and behavioral features) rather than something legally or ethically problematic, and provides a concrete audit artifact rather than a black-box justification.
• Using the same technique to catch feature leakage or unstable reliance: if a feature's SHAP contribution is unexpectedly large and doesn't have a plausible causal story (for example, a feature that could only be known AFTER the outcome was already determined), that's a strong, specific signal of leakage; similarly, if the model's SHAP-based feature importance shifts dramatically between similar time periods with no underlying business change, that instability itself is worth investigating as a reliability concern, since it suggests the model may be latching onto a spurious, unstable pattern rather than a genuinely predictive one.

Trade-offs and pitfalls
SHAP explanations are computed relative to the specific model and data it was trained on, so a large SHAP value confirms the MODEL is relying heavily on a feature, not that the feature is actually causally related to fraud, correlation and causation remain distinct even with a rigorous attribution method. It's also worth being upfront with stakeholders that SHAP values on correlated features can be unstable (the contribution can be arbitrarily split between two highly-correlated features depending on their exact values), which is a real limitation to disclose rather than presenting SHAP output as an unambiguous ground truth.

Direct answer
Sort all cases by predicted score, take the top k, and report the fraction of those top k that are actually true positives; this metric maps naturally onto a fraud review queue because a reviewer team with a fixed daily capacity effectively works exactly this way, they review the top-scored k cases each day, so precision at k directly measures how much of their limited attention is being well spent, rather than measuring performance at some abstract threshold the team may never actually operate at.

Structured elaboration and worked example (executed)

from typing import List

def precision_at_k(y_true: List[int], y_score: List[float], k: int) -> float:
    assert len(y_true) == len(y_score)
    assert 0 < k <= len(y_true)
    order = sorted(range(len(y_score)), key=lambda i: y_score[i], reverse=True)
    top_k = order[:k]
    hits = sum(y_true[i] for i in top_k)
    return hits / k

Executed on a small labeled example with 3 true positives out of 10 cases:

precision_at_k(k=1) = 1.0000
precision_at_k(k=3) = 1.0000
precision_at_k(k=5) = 0.6000
precision_at_k(k=10) = 0.3000

The top 3 scored cases (indices with scores 0.95, 0.9, and 0.85) were all 3 of the true fraud cases in this example, giving precision@3 of exactly 1.0. Expanding to the top 5 pulled in 2 more legitimate cases, dropping precision to 0.6, and at k=10 (the full set), precision necessarily converges to the overall base rate of positives, 3/10 = 0.3, which is a useful sanity check on the implementation: precision@k at k equal to the full dataset size must always equal the overall positive rate.

Complexity and edge cases
Sorting dominates the runtime at O(n log n) for n cases; a review-queue system reusing this same ranking every day at scale would typically maintain the scores in a pre-sorted or partially-sorted structure rather than re-sorting from scratch each time. Edge cases worth handling explicitly: k must be validated to be strictly positive and no larger than the number of cases available (both asserted above), and ties in score at the boundary of the top k need a deterministic tie-breaking rule (for example, sorting ties by transaction amount or timestamp) so the metric doesn't silently depend on an unstable sort order.

Trade-offs and pitfalls
Precision@k tells you nothing about recall, a model can have perfect precision@k while still missing the majority of total fraud that falls outside the queue's daily capacity, so it should always be reported alongside a recall or PR-AUC figure, not as a standalone success metric on its own.