Computer Vision Questions

Direct answer

Split by the unit that must be independent in deployment, not by the image file. If several photos can come from the same object, person, scene, camera session or video, put all of them on one side of the split (a grouped split), then stratify (keep class proportions similar) at the group level, and for anything ordered in time hold out the latest block. Find near-duplicates first and keep each cluster of them on one side of the split, split before any augmentation or normalization fitting, then freeze the test set and touch it once.

Terms. Leakage means information from the test set reaching training, so test scores look better than production will be. Stratification keeps each class's share roughly equal across splits. A group is a set of images that are not independent (same object photographed several ways, frames of one clip).

Why splitting by image fails

If object 17 has eight photos and a random split puts six in training and two in test, a model can pass the test by recognizing object 17, not the class. The score then measures memory of objects, but production will show it new objects.

Run it: the same data, two splits

The program builds 5 classes x 12 objects x 8 photos (each photo is a feature vector: the object's own offset from its class centre plus small per-photo noise; a stand-in for image embeddings). A 1-nearest-neighbour classifier is the simplest model that memorizes. Part two shows grouped plus stratified folds. Part three is 400 video frames with slow drift where neighbouring frames are near-duplicates.

import numpy as np
from sklearn.model_selection import train_test_split, GroupShuffleSplit, StratifiedGroupKFold
from sklearn.neighbors import KNeighborsClassifier

def make_data(seed, n_classes=5, objs_per_class=12, views=8, dim=20):
    r = np.random.default_rng(seed)
    centers = r.normal(0, 1.0, (n_classes, dim))
    X, y, obj = [], [], []
    for c in range(n_classes):
        for o in range(objs_per_class):
            base = centers[c] + r.normal(0, 1.5, dim)            # each object differs a lot from its class centre
            for v in range(views):                                # several photos of the same object
                X.append(base + r.normal(0, 0.2, dim)); y.append(c); obj.append(c * objs_per_class + o)
    return np.array(X), np.array(y), np.array(obj)

def no_group_overlap(g, tr, te):
    return len(set(g[tr]) & set(g[te])) == 0

rand_acc, grp_acc, leak_rand, leak_grp = [], [], [], []
for seed in range(10):
    X, y, g = make_data(seed)
    idx = np.arange(len(y))
    tr, te = train_test_split(idx, test_size=0.25, stratify=y, random_state=seed)     # naive: split photos
    clf = KNeighborsClassifier(1).fit(X[tr], y[tr]); rand_acc.append(clf.score(X[te], y[te]))
    leak_rand.append(not no_group_overlap(g, tr, te))
    tr2, te2 = next(GroupShuffleSplit(1, test_size=0.25, random_state=seed).split(X, y, g))   # split objects
    clf = KNeighborsClassifier(1).fit(X[tr2], y[tr2]); grp_acc.append(clf.score(X[te2], y[te2]))
    leak_grp.append(not no_group_overlap(g, tr2, te2))
print("1-nearest-neighbour test accuracy, mean over 10 seeds")
print(f"  split by photo (objects shared between train and test): {np.mean(rand_acc):.3f}  object overlap in {sum(leak_rand)}/10 seeds")
print(f"  split by object (no object in both sets):               {np.mean(grp_acc):.3f}  object overlap in {sum(leak_grp)}/10 seeds")

# stratified AND grouped folds: class balance per fold while keeping each object in one fold
X, y, g = make_data(0)
cv = StratifiedGroupKFold(n_splits=4, shuffle=True, random_state=0)
for k, (tr, te) in enumerate(cv.split(X, y, g)):
    print(f"fold {k}: test photos per class {np.bincount(y[te], minlength=5)}, objects shared with train: {len(set(g[tr]) & set(g[te]))}")

# temporal data: 400 consecutive video frames whose appearance drifts slowly; neighbouring frames are near-duplicates
res = {"random frames": [], "last 25% of time held out": []}; gaps = {k: [] for k in res}
for seed in range(20):
    r = np.random.default_rng(seed)
    drift = np.cumsum(r.normal(0, 0.3, (400, 6)), axis=0)
    t = np.arange(400); label = (np.sin(2 * drift[:, 0]) > 0).astype(int)      # a rule the early frames only partly cover
    Xf = drift + r.normal(0, 0.05, drift.shape)
    rt, re = train_test_split(t, test_size=0.25, random_state=seed)               # random frames
    bt, be = t[:300], t[300:]                                                       # earlier frames train, later frames test
    for name, a, b in (("random frames", rt, re), ("last 25% of time held out", bt, be)):
        res[name].append(KNeighborsClassifier(1).fit(Xf[a], label[a]).score(Xf[b], label[b]))
        gaps[name].append(np.min(np.abs(a[:, None] - b[None, :]), axis=0).mean())
for name in res:
    print(f"{name:28s} accuracy {np.mean(res[name]):.3f}, mean frames to nearest training frame {np.mean(gaps[name]):.1f} (20 seeds)")

Output:

1-nearest-neighbour test accuracy, mean over 10 seeds
  split by photo (objects shared between train and test): 1.000  object overlap in 10/10 seeds
  split by object (no object in both sets):               0.676  object overlap in 0/10 seeds
fold 0: test photos per class [24 24 24 24 24], objects shared with train: 0
fold 1: test photos per class [16 24 32 24 24], objects shared with train: 0
fold 2: test photos per class [16 40 16 24 24], objects shared with train: 0
fold 3: test photos per class [40  8 24 24 24], objects shared with train: 0
random frames                accuracy 0.862, mean frames to nearest training frame 1.1 (20 seeds)
last 25% of time held out    accuracy 0.537, mean frames to nearest training frame 50.5 (20 seeds)

Reading it:

• Splitting by photo scored 1.000 and every seed shared objects across the split; splitting by object scored 0.676 with zero shared objects. The 0.324 gap is leakage, not model quality, and no_group_overlap is the check that fails on the naive split. Run an assertion like it on every dataset build.
• With StratifiedGroupKFold no object appears on both sides (objects shared with train: 0), and class counts per fold are close but not equal (for example 16 to 40 photos of one class, because whole objects of 8 photos move together). Perfect balance and group integrity cannot both hold when groups are large; report the per-fold class counts and prefer group integrity. Hand check with the numbers above: each class has 12 objects of 8 photos, so a perfectly even 4-fold split would give every fold 3 objects of each class, which is 24 photos, and fold 0 shows exactly that. The counts 16 and 40 are 2 and 5 whole objects (2 x 8 and 5 x 8). The splitter can only move whole objects, and when it cannot give every class 3 objects in every fold at once it settles for a compromise across the classes, so one class can land with 2 objects in one fold and 5 in another.
• For ordered data, random frames scored 0.862 against 0.537 when the final 25% of time was held out, and the nearest training frame was 1.1 frames away for random splits against 50.5 for the time split (20 seeds). The toy label rule here is built so that unseen later regions are not covered by early ones, so the size of the gap is a property of this simulation; the direction is the point: a random split of correlated frames mostly tests interpolation (predicting between points the model has seen, here neighbouring frames), and deployment is extrapolation (predicting beyond them, here forward in time). How large the gap is on real video depends on how fast the scene changes and how long the held-out block is, so measure it on your own data: score the same model under both splits and report both numbers, because the difference is the size of the leak.

Practical checklist

1. Find the real groups. Object or product id, patient, user, camera/device, scene, shooting session, video, location. When ids are missing, build groups by clustering embeddings or perceptual hashes and keep clusters whole.
2. Remove near-duplicates across splits. Exact duplicates are found with a file hash (a short fingerprint of the file's bytes). Near-duplicates need a fingerprint of the picture's content: a perceptual hash is a short code that stays nearly the same when an image is resized or recompressed, so two codes that differ in only a few bits mean the same photo. Alternatively compare image embeddings (vectors produced by a pretrained network) with cosine similarity (1.0 means the two vectors point the same way) and call a pair duplicates above a threshold chosen by looking at example pairs on both sides of it. Scraped data often has the same photo at several resolutions.
3. Stratify at the group level (class labels, and for detection the rare classes or object-size buckets) so rare classes appear in validation and test.
4. Time: train on the past, validate on a later block, test on the latest. For videos split by clip, never by frame. If seasonal, make sure each split covers the same seasonal range or test on a whole later season.
5. Fit preprocessing on training data only. Anything computed from data is a fitted step. For example, the per-channel mean and standard deviation used to normalize pixels must come from training images only; computing them over all images lets test-image brightness shape the numbers the model trains on. The same holds for a learned encoder (a network trained to turn images into vectors, for example for clustering or de-duplication) and for any list built from the data, such as the most frequent labels or tags. Apply augmentation only to training images and after the split; an augmented copy of a test image in training is leakage.
6. Three roles, three sets. Training fits weights; validation picks hyperparameters (settings you choose before training, such as learning rate or model size) and when to stop; the test set is read once for the final number. Looking at test results and then changing the model turns the test set into a validation set.
7. Match the test set to deployment. If production images come from new cameras or sites, hold out entire cameras or sites. Report results by site as well as overall.
8. Small datasets: use grouped k-fold cross-validation and report the spread across folds.

Mistakes

• Stratifying by class but not by group.
• Deduplicating after the split, which can leave twins in both sets.
• Splitting video frames at random and reporting 99%.
• Reusing the test set for model selection across many experiments.

The idea in plain words

A pretrained backbone is the body of a network (the stack of layers that turns an image into a feature vector) that was already trained on a large image collection. Transfer learning reuses it, because the early layers learn generic things (edges, textures, simple shapes) that almost any image task needs. There are two ways to reuse it:

• Feature extraction: freeze the backbone (its weights never change) and train only a new final layer, the head, that maps the feature vector to your classes. The backbone is a fixed function; you learn a small classifier on top of its outputs.
• Full fine-tuning: start from the pretrained weights, replace the head, and let all layers keep learning, with a small learning rate so the useful pretrained features are adjusted rather than destroyed.

Feature extraction has few trainable parameters, so it is cheap, stable and hard to overfit; fine-tuning has many, so it can adapt the features to your images but can overfit a small dataset and forget what it started with.

Side by side

A middle path is common: train the head first, then unfreeze only the last blocks (the layers closest to the output, which are the most task-specific), keeping the early generic layers frozen.

What each strategy is allowed to change, with counted numbers

No accuracy race between the strategies is shown here, because no pretrained ResNet is available to the program. A race on a small stand-in network pretrained on a few hundred tiny digit images would say little about a real backbone pretrained on a large, diverse image collection, and in such a toy training from scratch can tie the transfer methods, which is not what is seen in practice. What is widely observed in practice, and the reason to start with feature extraction, is that with a few hundred to a few thousand labelled images a pretrained backbone usually beats training from scratch by a clear margin. The advantage shrinks as the target dataset and the training time grow: a published study of detection and segmentation (He, Girshick and Dollar, 2018) found that pretraining mainly speeds up early convergence and does not necessarily improve final accuracy once the target data is large, which is why the from-scratch comparison in step 7 is only worth running when you have the budget. What the program below does show is the counted reason (how many numbers each strategy has to fit from your labels) and one trap (BatchNorm: a layer that normalises activations using the mean and spread of the current batch during training, and keeps running averages of them for use at inference). The network is a tiny stand-in with the same structure as a real one: a backbone ending in a feature vector, plus a head.

import torch
import torch.nn as nn
from sklearn.datasets import load_digits

torch.set_num_threads(1)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32).reshape(-1, 1, 8, 8)

class Net(nn.Module):
    def __init__(self, n_out):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(),
            nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.AdaptiveAvgPool2d(2), nn.Flatten())            # 32 * 2 * 2 = 128 features
        self.head = nn.Linear(128, n_out)                     # the new final layer: 128 -> 2 classes
    def forward(self, x):
        return self.head(self.backbone(x))

def n_trainable(m):
    return sum(p.numel() for p in m.parameters() if p.requires_grad)

# 1. Count what each strategy is allowed to change, and what that means per labelled image.
model = Net(2)
all_params = n_trainable(model)
model.backbone.requires_grad_(False)                          # feature extraction: freeze the backbone
head_params = n_trainable(model)
print(f"trainable parameters: head only = {head_params}, all layers = {all_params}")
for n_labels in (20, 2000):
    print(f"  with {n_labels:>4} labelled images: head only = {head_params / n_labels:6.1f} parameters per image, all layers = {all_params / n_labels:6.1f}")

# 2. BatchNorm: frozen weights do not freeze the running statistics unless the layer is in eval mode.
def bn_drift(put_in_eval):
    torch.manual_seed(0)
    net = Net(2)
    net.backbone.requires_grad_(False)
    before = net.backbone[1].running_mean.clone()
    net.train()
    if put_in_eval:
        net.backbone.eval()
    net(X[:64] * 3.0 + 1.0)                                   # a batch whose statistics differ from the default
    return (net.backbone[1].running_mean - before).abs().max().item()

print(f"BatchNorm running_mean change with frozen weights, backbone in train mode: {bn_drift(False):.4f}")
print(f"BatchNorm running_mean change with frozen weights, backbone in eval mode : {bn_drift(True):.4f}")

Output:

trainable parameters: head only = 258, all layers = 5154
  with   20 labelled images: head only =   12.9 parameters per image, all layers =  257.7
  with 2000 labelled images: head only =    0.1 parameters per image, all layers =    2.6
BatchNorm running_mean change with frozen weights, backbone in train mode: 0.2160
BatchNorm running_mean change with frozen weights, backbone in eval mode : 0.0000

Reading the output:

• Parameters: a frozen backbone leaves 258 trainable parameters (128 features times 2 classes, plus 2 biases) against 5154 when everything trains. With 20 labelled images that is about 13 parameters per image for the head against about 258 per image for full fine-tuning; with 2000 images it is 0.1 against 2.6. Many parameters per labelled image is the situation in which a model can memorise the training set, which is why feature extraction is the safe start on tiny data and fine-tuning becomes reasonable as labels grow.
• BatchNorm: with every weight frozen, the layer's running mean still moved by 0.2160 when the backbone ran in training mode, and by 0.0000 in evaluation mode. Training a head on a small, differently distributed dataset with BatchNorm in training mode silently rewrites the backbone's statistics. Put the frozen backbone in evaluation mode.
• Learning rates, with example values: the learning rate is the size of the step the optimiser takes along the gradient on each update. A new head starts random and must move far, so give it a normal rate such as 1e-3; the pretrained backbone is already good and only needs nudging, so give it 10 to 100 times less, such as 1e-4 to 1e-5. These are starting values to validate on your data, not measured optima.

Applying a pretrained ResNet to a small labelled dataset: the steps

A ResNet is a family of CNNs built from residual blocks (blocks whose input is added back to their output, which makes deep networks trainable). The steps do not depend on the depth.

1. Use the weights' own preprocessing: the input size, channel order and per-channel mean and standard deviation the weights were trained with. A mismatch here silently lowers accuracy and raises no error.
2. Replace the head: swap the final fully connected layer for a new one with one output per class (read the backbone's feature width from the layer itself rather than assuming it).
3. Baseline first (feature extraction): freeze the backbone, put it in evaluation mode, train the head. This takes minutes, and its validation score is the number to beat.
4. Fine-tune if the baseline leaves room: unfreeze the last block or two, then more if data allows. Give the head a normal learning rate and the unfrozen backbone a learning rate roughly 10 to 100 times smaller (a starting range to validate, not a rule). A short warm-up (start with a tiny learning rate and ramp up over the first steps so the random head does not push big gradients into the backbone) and a decaying schedule (lower the rate gradually as training goes on) help.
5. Augment and regularise: flips and crops that suit the data, then regularisers (techniques that stop the model memorising): weight decay (a small penalty that pulls weights toward zero), label smoothing (training against softened targets, for example 0.9 on the correct class with the remaining 0.1 spread over the other classes, instead of a hard 1 and 0, so the model is not pushed to be over-confident), or dropout (randomly zeroing some activations during training) in the head; stop on validation loss. On a small dataset the augmentation and the validation split matter most; treat the regularisers as extras to add one at a time if validation shows overfitting.
6. Validate honestly: a stratified split (each class keeps its share), cross-validation (repeating the train and validate cycle on several different splits and averaging) when the set is tiny, and check that near-duplicate images do not sit on both sides of the split.
7. Compare against training from scratch only if you have the budget; with a few hundred images it rarely wins.

Trade-offs and mistakes

• Fine-tuning everything at the head's learning rate wrecks the pretrained features in the first few steps.
• Forgetting the BatchNorm mode, as measured above.
• Reporting accuracy from a split where near-duplicates leak across train and validation.
• Choosing by training accuracy instead of validation.

The idea in plain words

A few hundred labelled images are too few to learn visual features from scratch, but large generic image collections have already taught a backbone (the feature-extracting part of a network, everything before the final classification layer) what edges, textures, shapes and parts look like. Transfer learning reuses that: you keep the backbone and train only what is specific to your problem. The strategy is to start with the cheapest version (frozen backbone features plus a simple classifier on top), measure it honestly, and move to partial fine-tuning only if the measurement says there is headroom. What can make it fail is mostly not the model; it is a domain gap, an evaluation that leaks, and shortcuts in the data.

The strategy, in order

1. Fix the evaluation before touching a model. With a few hundred images, the validation protocol decides whether you can trust any result. Define the unit of generalisation (a new product, a new photo session, a new store) and split by it, so that images of the same physical item never appear on both sides. Use k-fold cross-validation (split the data into k parts, train on k-1, test on the remaining one, rotate) with that grouping, report per-class recall (for each class separately, the share of its images the model found, so a rare class cannot hide behind a big one), and keep a final test set you look at once.
2. Frozen features plus a linear classifier (a linear probe). Pass each image through the pretrained backbone once, store the feature vector (an embedding, a fixed-length list of numbers summarising the image), and train a regularised logistic regression on top (logistic regression is a linear classifier that outputs class probabilities; regularised means it is penalised for large weights, which curbs overfitting when examples are few; the setting C is the inverse of that penalty, so a small C pulls the weights harder towards zero, and a code example below compares three values). It trains in seconds, has few parameters to overfit, and is a strong baseline. Compare two or three candidate backbones this way, choosing the one whose features separate your classes best on grouped cross-validation, not the one with the best name or generic benchmark score.
3. Partial fine-tuning if the probe leaves headroom (headroom means a gap between the probe's score and the score you need). Fine-tuning means continuing to train the pretrained weights on your data; here you unfreeze the last block or two of the backbone (let those layers learn again while the earlier ones stay fixed), use a small learning rate for them (and a larger one for the new classifier), apply label-preserving augmentation (crops, flips if the product has no handedness, colour jitter within what the real cameras produce), stop early on grouped validation, and compare to the probe on the same folds. With a few hundred images, full fine-tuning at a high rate tends to erase the pretrained features; if it does not beat the probe on grouped folds, ship the probe.
4. Get more signal cheaply. Writing down labelling rules for look-alike classes and labelling more images is the core move. Choose which images to label next with active learning (the model picks which unlabelled images a person should label next). The common rule, most uncertain first, tends to suit larger labelling budgets; at a budget of a few hundred labels, published work on low-budget active learning (Hacohen, Dekel and Weinshall, 2022, "Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets") finds that typical, representative examples are the better ones to query. So compare any selection rule against random picks on grouped validation before trusting it. Two heavier options come last and are rarely the first thing to try with a few hundred images: pseudo-labels (the model's own confident predictions on unlabelled images used as extra training labels, which can reinforce its mistakes) and self-supervised adaptation of the backbone (continuing to train it on your unlabelled product images with a label-free task). Try them only if steps 1 to 3 plateau and you have a large unlabelled pool.
5. Plan for abstention. A niche product classifier will see items from no known class. Abstention means the model declines to answer: choose a confidence threshold on validation data (the minimum predicted probability for the top class below which the image goes to a person), rather than forcing a label. The threshold trades coverage for accuracy, and the code example below prints that trade-off.

Worked example: why step 1 comes first

The program below uses a stand-in for backbone features: random vectors built as category signal plus item-specific appearance plus photo noise, for 3 categories with 20 physical items each and 5 photos per item (300 images). A real system would take these vectors from a pretrained backbone; here the point is the evaluation, not the features. The classifier is logistic regression with C=1.0. It reports accuracy under three protocols averaged over 10 generated datasets: a random 5-fold split, a split that keeps every item's photos together, and the grouped split with a shared offset added to every test embedding to imitate a new camera or background.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, StratifiedGroupKFold

# STAND-IN for frozen-backbone embeddings: 3 product categories, 20 physical items per category, 5 photos per item.
# embedding = category signal + item-specific appearance + photo noise. A real system would use a pretrained backbone's features.
D, CLASSES, ITEMS, PHOTOS = 32, 3, 20, 5

def make(seed):
    rng = np.random.default_rng(seed)
    cmean = rng.standard_normal((CLASSES, D)) * 0.35
    X, y, g = [], [], []
    for c in range(CLASSES):
        for i in range(ITEMS):
            item = rng.standard_normal(D) * 1.0                    # what makes THIS item look like itself (shared by its photos)
            for _ in range(PHOTOS):
                X.append(cmean[c] + item + 0.5 * rng.standard_normal(D))
                y.append(c); g.append(c * ITEMS + i)
    return np.array(X), np.array(y), np.array(g), cmean

def cv_acc(X, y, splits, shift=None):
    accs = []
    for tr, te in splits:
        Xte = X[te] + (shift if shift is not None else 0)
        clf = LogisticRegression(C=1.0, max_iter=2000).fit(X[tr], y[tr])
        accs.append(clf.score(Xte, y[te]))
    return float(np.mean(accs))

print("300 images (3 categories x 20 items x 5 photos); linear probe on stand-in embeddings; 5-fold; mean over 10 datasets")
rows = {"random split": [], "split by item": [], "split by item + camera shift": []}
for seed in range(10):
    X, y, g, _ = make(seed)
    rand = list(StratifiedKFold(5, shuffle=True, random_state=0).split(X, y))
    grp = list(StratifiedGroupKFold(5, shuffle=True, random_state=0).split(X, y, g))
    rows["random split"].append(cv_acc(X, y, rand))
    rows["split by item"].append(cv_acc(X, y, grp))
    shift = np.random.default_rng(1000 + seed).standard_normal(D) * 0.8   # a new camera/background adds a shared offset to every test embedding
    rows["split by item + camera shift"].append(cv_acc(X, y, grp, shift))
for k, v in rows.items():
    print(f"{k:30s} accuracy mean {np.mean(v):.3f}  (min {np.min(v):.3f}, max {np.max(v):.3f})")
print("chance level: 0.333")
# does any item appear on both sides of the split?
X, y, g, _ = make(0)
leak = lambda splits: sum(len(set(g[tr]) & set(g[te])) for tr, te in splits)
print("items present in both train and test folds, summed over 5 folds -> random:", leak(list(StratifiedKFold(5, shuffle=True, random_state=0).split(X, y))),
      "| grouped:", leak(list(StratifiedGroupKFold(5, shuffle=True, random_state=0).split(X, y, g))))

Output:

300 images (3 categories x 20 items x 5 photos); linear probe on stand-in embeddings; 5-fold; mean over 10 datasets
random split                   accuracy mean 0.903  (min 0.800, max 0.953)
split by item                  accuracy mean 0.639  (min 0.503, max 0.797)
split by item + camera shift   accuracy mean 0.611  (min 0.470, max 0.773)
chance level: 0.333
items present in both train and test folds, summed over 5 folds -> random: 206 | grouped: 0

The same data and the same classifier score 0.903 under a random split and 0.639 when items are kept together; chance is 0.333. The gap is leakage: in the random split, 206 item-fold pairs have an item on both sides (the sum over 5 folds, for the first dataset), so the classifier memorises item appearance instead of category. The shift to a new camera lowers the grouped score further, to 0.611. The spread across datasets (0.503 to 0.797 for the grouped split) also shows how noisy a score from 300 images is, which is why you repeat or cross-validate before deciding. These numbers describe this stand-in and its settings only; they show the direction and the size of the trap, not what a real dataset would score.

The pipeline in code: embeddings, probe, grouped folds, abstention

The worked example above shows the evaluation trap. This program shows the strategy itself, end to end. The images and the backbone are both stand-ins: the images are 12 by 12 synthetic arrays and the "backbone" is a small untrained convolutional network, so its features are random; a real system would load a pretrained backbone here and feed it real photos, and the steps after the embeddings would be unchanged. It extracts one embedding per image with gradients off, compares three values of C on grouped 5-fold cross-validation (on the first dataset, then on ten others), collects an out-of-fold confidence for each image, and prints what each abstention threshold costs and buys.

import numpy as np
import torch
from torch import nn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedGroupKFold
torch.set_num_threads(1)

# The pipeline of steps 1 and 2: extract embeddings with a frozen backbone, fit a linear probe, grouped cross-validation, abstention threshold.
# STAND-INS: the "images" are 12 x 12 synthetic arrays and the "backbone" is a small UNTRAINED convolutional network, so the features
# are random. A real system would load a pretrained backbone here and feed it real photos; nothing below depends on that choice.
CLASSES, ITEMS, PHOTOS = 3, 20, 5

def make_embeddings(seed):
    """Synthetic images (class prototype + item appearance + photo noise) and their frozen-backbone embeddings."""
    rng = np.random.default_rng(seed)
    protos = 0.7 * rng.standard_normal((CLASSES, 12, 12))
    imgs, y, g = [], [], []
    for c in range(CLASSES):
        for i in range(ITEMS):
            item = 1.0 * rng.standard_normal((12, 12))                 # appearance shared by the photos of one physical item
            for _ in range(PHOTOS):
                imgs.append(protos[c] + item + 0.6 * rng.standard_normal((12, 12))); y.append(c); g.append(c * ITEMS + i)
    X_img = torch.tensor(np.stack(imgs), dtype=torch.float32)[:, None]
    torch.manual_seed(seed)
    backbone = nn.Sequential(nn.Conv2d(1, 8, 3), nn.ReLU(), nn.AdaptiveAvgPool2d(2), nn.Flatten())   # frozen: no optimiser ever sees it
    backbone.eval()
    with torch.no_grad():                                              # no gradients: the backbone is only a feature extractor
        emb = backbone(X_img).numpy()                                  # (300 images, 32 numbers each): one embedding per image
    return emb, np.array(y), np.array(g)

emb, y, g = make_embeddings(0)
print("embeddings:", emb.shape)

folds = list(StratifiedGroupKFold(5, shuffle=True, random_state=0).split(emb, y, g))   # an item's photos never straddle the split
for C in (0.01, 1.0, 100.0):                                       # C is the INVERSE regularisation strength: small C = stronger pull towards small weights
    accs = [LogisticRegression(C=C, max_iter=5000).fit(emb[a], y[a]).score(emb[b], y[b]) for a, b in folds]
    print(f"C={C:<6} grouped 5-fold accuracy {np.mean(accs):.3f}")

# the same comparison on 10 other generated datasets, so the ordering of the C values is not read off one draw
means = {C: [] for C in (0.01, 1.0, 100.0)}
for seed in range(1, 11):
    e2, y2, g2 = make_embeddings(seed)
    f2 = list(StratifiedGroupKFold(5, shuffle=True, random_state=0).split(e2, y2, g2))
    for C in means:
        means[C].append(np.mean([LogisticRegression(C=C, max_iter=5000).fit(e2[a], y2[a]).score(e2[b], y2[b]) for a, b in f2]))
for C, v in means.items():
    print(f"C={C:<6} over 10 other datasets: mean {np.mean(v):.3f}, min {np.min(v):.3f}, max {np.max(v):.3f}")
print("datasets where C=100.0 beats C=1.0:", int(np.sum(np.array(means[100.0]) > np.array(means[1.0]))), "of 10; where C=0.01 is the lowest:",
      int(np.sum((np.array(means[0.01]) < np.array(means[1.0])) & (np.array(means[0.01]) < np.array(means[100.0])))), "of 10")

# out-of-fold confidence for every image (each image is scored by a probe that never saw its item), with C = 1.0
conf = np.zeros(len(y)); correct = np.zeros(len(y), bool)
for a, b in folds:
    probe = LogisticRegression(C=1.0, max_iter=5000).fit(emb[a], y[a])
    conf[b] = probe.predict_proba(emb[b]).max(1); correct[b] = probe.predict(emb[b]) == y[b]
p, n = correct.mean(), len(y)
print(f"pooled out-of-fold accuracy {p:.3f} on {n} images; standard error sqrt(p(1-p)/n) = {np.sqrt(p * (1 - p) / n):.3f}")
print("abstention threshold -> share of images answered, accuracy on the answered ones")
for t in (0.0, 0.5, 0.6, 0.7):
    keep = conf >= t
    print(f"  confidence >= {t}: answered {keep.mean():.2f}, accuracy {correct[keep].mean():.3f}")
print("for comparison: accuracy 0.60 measured on 150 images has standard error", round(float(np.sqrt(0.6 * 0.4 / 150)), 3))

Output:

embeddings: (300, 32)
C=0.01   grouped 5-fold accuracy 0.610
C=1.0    grouped 5-fold accuracy 0.700
C=100.0  grouped 5-fold accuracy 0.693
C=0.01   over 10 other datasets: mean 0.732, min 0.640, max 0.920
C=1.0    over 10 other datasets: mean 0.783, min 0.673, max 0.950
C=100.0  over 10 other datasets: mean 0.821, min 0.693, max 0.970
datasets where C=100.0 beats C=1.0: 9 of 10; where C=0.01 is the lowest: 10 of 10
pooled out-of-fold accuracy 0.700 on 300 images; standard error sqrt(p(1-p)/n) = 0.026
abstention threshold -> share of images answered, accuracy on the answered ones
  confidence >= 0.0: answered 1.00, accuracy 0.700
  confidence >= 0.5: answered 0.70, accuracy 0.789
  confidence >= 0.6: answered 0.39, accuracy 0.846
  confidence >= 0.7: answered 0.21, accuracy 0.887
for comparison: accuracy 0.60 measured on 150 images has standard error 0.04

Reading it: each image becomes 32 numbers. On the first dataset C=1.0 and C=100.0 are about equal (0.700 and 0.693) and a very small C is lower (0.610). Over ten other generated datasets the smallest C is the lowest every time and C=100.0 beats C=1.0 in 9 of 10 (means 0.821 and 0.783), so which penalty strength wins depends on the draw of data, and the first dataset alone would have suggested a tie. Treat the penalty strength as a setting to check on grouped folds rather than assume; the features here are random, so which C wins on real embeddings is a different matter. To choose the abstention threshold, pick the coverage you can afford: at a confidence of 0.6 the probe answers 39 percent of images and is right on 84.6 percent of them, while the rest go to a person. The standard error of an accuracy $p$ measured on $n$ images is $\sqrt{p(1-p)/n}$: for 0.700 on 300 images it is 0.026, and an accuracy of 0.60 measured on 150 images has a standard error of $\sqrt{0.6 \times 0.4 / 150} = 0.04$, so a 0.62 score on such a set is well inside the noise of a 0.60 one. This standard error treats images as independent; photos of one item are correlated, so the real uncertainty is larger, which is one more reason a 0.60 vs 0.62 difference is not a reason to prefer the heavier model.

What can make it fail

• Domain gap. If the niche products look nothing like the generic images (industrial parts, macro shots, microscopy, textures that differ by tiny details), the backbone's features may not separate the classes. Symptom: the probe sits near the accuracy of a trivial baseline and fine-tuning barely moves it. Response: fine-tune deeper layers, try a backbone pretrained on something closer, or collect more labels.
• Leaky validation (shown above): near-duplicate photos, frames from one video, same item in many views. Check by grouping and by counting overlap.
• Shortcut features. The classifier may use the background, the lighting, the watermark or the shelf instead of the product. Check by testing on images with a different background, by masking the object or the background and seeing which one the score depends on, and by looking at the images the model gets wrong.
• Class imbalance and fine-grained classes. Report per-class recall, not only accuracy; a majority-class guess can look good.
• Label noise. With few images a handful of wrong labels matters; review the most confident mistakes.
• Deployment shift. New cameras, seasons, packaging redesigns. Keep a small labelled sample from production and re-measure on it.
• Licensing or provenance of the pretrained weights, for commercial use.

Common mistakes

Tuning many things on the same few hundred validation images, so the chosen configuration is the one that fits their noise; reporting accuracy from a random split; fine-tuning everything at the rate used for training from scratch; and forgetting that the "known classes only" assumption fails the day an unknown product shows up.

The idea in plain words

When you fine-tune a pretrained image classifier, the backbone (the feature extractor, trained earlier on a large dataset) already knows how to describe images. What the few minority examples must teach is only where the decision boundaries (the places in feature space where the predicted class switches) go. Two things go wrong: the classification head (the last layer that turns features into class scores) learns the class frequencies as a prior (the share of the training images each class has, which the head absorbs as a built-in lean towards common classes), so it leans towards the majority classes; and with a handful of images the network can memorise instead of generalise. The fixes act at four places, which are the four the question names: sampling (what the model sees), loss (how errors are weighted), augmentation (how few images are stretched), and metrics (how you judge). The metric comes first, because plain accuracy can hide the problem when the test traffic is skewed (a computed example below shows how).

What I would change

1. Metrics. Evaluate on a validation set that has enough minority examples to be meaningful (if you must, build it with a minimum number per class and keep it out of training). Report per-class recall, macro-F1 (the F1 score of each class, averaged with equal weight per class), and the confusion matrix (a table whose rows are the true class and whose columns are the predicted class, so a class that is never predicted shows up as an empty column); accuracy can look excellent while a class is never predicted. Use the same decision-threshold policy you will deploy (the rule that turns scores into answers, for example a minimum confidence below which the system abstains).
2. Training strategy for few examples. Start by training only the head on frozen features (a linear probe: one linear layer on top of features that do not change), which has the fewest parameters to overfit. If that leaves accuracy on the table, unfreeze the top layers (let the last blocks of the backbone learn again instead of staying fixed) with a small learning rate, or all layers with a smaller learning rate for the backbone than for the head. Stop early on a validation metric that cares about the minority (macro-F1), not on training loss.
3. Sampling. Draw minority images more often (class-balanced sampling gives every class equal draw probability; square-root sampling draws each class in proportion to the square root of its size, a middle path, so classes of 400 and 4 images are drawn in the ratio 20 to 2, which is 10 to 1 instead of 100 to 1) rather than only collecting duplicates offline. It is the cheapest change but it repeats the same few images, so it needs augmentation to avoid memorisation.
4. Loss. Weight each class's loss inversely to its frequency, or adjust the logits (the raw class scores before softmax) after training by subtracting the log of the class prior. The reason it works: a head trained on skewed data scores common classes higher partly just because they were common, and subtracting the log prior removes that built-in lean so the classes compete as if they had been equally frequent. A numeric trace follows the measured comparison. Weighted losses and sampling do overlapping jobs; use one of them as the main tool and test the other, because doing both at full strength over-corrects.
5. Augmentation. Apply stronger augmentation to minority images (flips where the class allows it, small shifts and rotations, crops, colour jitter, and mixing methods that blend images, for example mixup, which blends two images and their labels with a random mixing weight), but only transformations that keep the label true. For example a flipped "6" is not a "6" in digit images.
6. Data. Check whether the minority labels are right, and get more minority images if you can (targeted collection or active learning, where the model suggests which unlabelled images to label next). Nothing here replaces new information.

Where I would start. First fix the metrics, then train the head alone with no imbalance fix to get a baseline. The first imbalance tool I would try is post-hoc logit adjustment (it needs no retraining, and it was the strongest of the four methods measured below) when deployment is balanced, or class weights when it is not; I would add oversampling with augmentation only if minority recall is still short, because it repeats the same few images. The measured comparison tests the methods one at a time and never combines them, so "use one as the main tool" is a caution about over-correcting, not a measured result.

A measured comparison

The program uses the 10-class scikit-learn digits images (8 by 8 pixels). The "backbone" is a stand-in: a fixed random projection followed by a ReLU, never trained, standing in for a pretrained network; a real system would put a pretrained image network there, and only the head is trained here. The classes 7, 8 and 9 keep only 6 training examples each, while the other classes keep about 107. The test set is balanced. It compares four ways of training the head, averaging over 10 different random splits because one split cannot show that one method beats another.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, recall_score
from sklearn.model_selection import train_test_split

# STAND-IN for a pretrained backbone: a fixed random-projection + ReLU feature map that is never trained.
# A real system would put a pretrained image network here. Only the classification head is trained.
X, y = load_digits(return_X_y=True)
X = X / 16.0
W = np.random.default_rng(0).normal(0, 1, (64, 256)) / 8.0
feat = lambda A: np.maximum(A @ W, 0)

MINOR = [7, 8, 9]                                   # minority classes
def run(seed, method):
    rng = np.random.default_rng(seed)
    Xa, Xb, ya, yb = train_test_split(X, y, test_size=0.4, random_state=seed, stratify=y)
    keep = np.ones(len(ya), bool)
    for c in MINOR:                                  # keep only 6 training examples of each minority class
        idx = np.where(ya == c)[0]; keep[idx[6:]] = False
    Xa, ya = Xa[keep], ya[keep]
    if method == "oversample + shift augmentation":  # repeat minority images up to the majority size, each with a random 1 px shift
        cnt = np.bincount(ya).max(); xs, ys = [Xa], [ya]
        for c in MINOR:
            base = Xa[ya == c]
            for _ in range(cnt - len(base)):
                im = base[rng.integers(len(base))].reshape(8, 8)
                im = np.roll(im, tuple(rng.integers(-1, 2, 2)), (0, 1)); xs.append(im.reshape(1, -1)); ys.append([c])
        Xa, ya = np.vstack(xs), np.concatenate([np.asarray(a).ravel() for a in ys])
    cw = "balanced" if method == "class-weighted loss" else None
    clf = LogisticRegression(max_iter=2000, C=1.0, class_weight=cw).fit(feat(Xa), ya)
    logit = clf.decision_function(feat(Xb))
    if method == "logit adjustment (post hoc)":      # subtract log prior of the TRAIN labels from each class score
        prior = np.bincount(ya, minlength=10) / len(ya); logit = logit - np.log(prior)
    pred = clf.classes_[logit.argmax(1)]
    return [(pred == yb).mean(), f1_score(yb, pred, average="macro"), recall_score(yb, pred, labels=MINOR, average="macro")]

print("train: majority classes ~%d examples each, classes 7/8/9 have 6 each; test is balanced" % (len(y) * 0.6 / 10))
print("mean over 10 splits (std of minority recall in brackets)")
print("%-34s %8s %9s %16s" % ("method", "accuracy", "macro-F1", "minority recall"))
for m in ["plain", "class-weighted loss", "oversample + shift augmentation", "logit adjustment (post hoc)"]:
    R = np.array([run(s, m) for s in range(10)])
    print("%-34s %8.3f %9.3f %9.3f (%.3f)" % (m, R[:, 0].mean(), R[:, 1].mean(), R[:, 2].mean(), R[:, 2].std()))

The complete output:

train: majority classes ~107 examples each, classes 7/8/9 have 6 each; test is balanced
mean over 10 splits (std of minority recall in brackets)
method                             accuracy  macro-F1  minority recall
plain                                 0.829     0.806     0.457 (0.055)
class-weighted loss                   0.914     0.910     0.756 (0.057)
oversample + shift augmentation       0.888     0.880     0.688 (0.038)
logit adjustment (post hoc)           0.935     0.935     0.884 (0.032)

Reading the output:

• Plain training reaches 0.829 accuracy but only 0.457 recall on the three minority classes: the head has learned to prefer the majority classes. This test set is balanced, so accuracy (0.829) and macro-F1 (0.806) sit close together here; the skewed-test example after this list shows where accuracy hides the problem.
• Class weighting raises minority recall to 0.756 and macro-F1 from 0.806 to 0.910. Oversampling with a 1-pixel shift augmentation gets 0.688 and 0.880. Post-hoc logit adjustment gets 0.884 and 0.935, the best here. Every other method beats plain training by at least 0.231 in minority recall, far more than the split-to-split standard deviation (at most 0.057). Logit adjustment leads class weighting by 0.128, more than twice that spread, while the 0.068 gap between weighting and oversampling is close to it, so read the lead of logit adjustment as real in this setup and that pair as not ranked.
• This test set is balanced, and logit adjustment subtracts the training prior, which is right when deployment is balanced. If deployment has the same skew as training, adjusting hurts the overall error; then choose the operating point by the cost of each error type instead.
• The result is for a linear head on fixed random features and 8 by 8 images. It shows mechanisms; it does not predict the gain on your fine-tuned network. A network that memorises its few minority images is already over-confident on them, so a post-hoc shift fitted to the training prior can correct less than it does here; the same frequency shift can also be built into the loss during training, and the logit-adjustment literature presents both forms.

Why accuracy alone misleads, and the logit adjustment traced by hand

The comparison above uses a balanced test set, so it cannot show accuracy hiding anything. Real traffic is usually skewed. The program below takes 1000 test images (970 of a common class, 20 and 10 of two rare classes) and a model that predicts the common class for everything, and prints the metrics. Then it traces logit adjustment on one image: training was 90 percent class A and 10 percent class B, and the head's raw scores favour A.

import numpy as np
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, recall_score

# Part 1: on a test set that is skewed like real traffic, a model that never predicts the two rare classes still scores high accuracy.
y_true = np.array([0] * 970 + [1] * 20 + [2] * 10)      # 970 common, 20 + 10 rare images
y_pred = np.zeros(1000, dtype=int)                       # predicts the common class for everything
print("accuracy       :", round(accuracy_score(y_true, y_pred), 3))
print("per-class recall:", np.round(recall_score(y_true, y_pred, average=None, zero_division=0), 3).tolist())
print("macro-F1       :", round(f1_score(y_true, y_pred, average="macro", zero_division=0), 3))
print("confusion matrix (rows = true class, columns = predicted class):")
print(confusion_matrix(y_true, y_pred))

# Part 2: logit adjustment on one image. Training had 90 percent class A and 10 percent class B.
prior = np.array([0.90, 0.10])
logits = np.array([2.0, 1.0])                            # raw scores from the head: A looks better
softmax = lambda z: np.exp(z) / np.exp(z).sum()
adjusted = logits - np.log(prior)                        # subtract the log of the training prior from each class score
print("log prior          :", np.round(np.log(prior), 3).tolist())
print("raw scores         :", logits.tolist(), "-> probabilities", np.round(softmax(logits), 3).tolist(), "-> predicts", "AB"[logits.argmax()])
print("adjusted scores    :", np.round(adjusted, 3).tolist(), "-> probabilities", np.round(softmax(adjusted), 3).tolist(), "-> predicts", "AB"[adjusted.argmax()])

Output:

accuracy       : 0.97
per-class recall: [1.0, 0.0, 0.0]
macro-F1       : 0.328
confusion matrix (rows = true class, columns = predicted class):
[[970   0   0]
 [ 20   0   0]
 [ 10   0   0]]
log prior          : [-0.105, -2.303]
raw scores         : [2.0, 1.0] -> probabilities [0.731, 0.269] -> predicts A
adjusted scores    : [2.105, 3.303] -> probabilities [0.232, 0.768] -> predicts B

Reading it: 970 of 1000 images are right, so accuracy is 0.97, yet recall on both rare classes is 0 and macro-F1 is 0.328; the confusion matrix shows two empty columns. In the trace, the raw scores 2.0 and 1.0 give A a probability of 0.731. The log prior is $\ln 0.9 = -0.105$ for A and $\ln 0.1 = -2.303$ for B. Subtracting it raises B's score by 2.303 and A's by only 0.105, so the adjusted scores are 2.105 and 3.303, and B wins with probability 0.768. The head's evidence for A was mostly "A is common"; removing that lets the image's own evidence decide.

Common mistakes

• Reporting accuracy alone: on a skewed test set a model that never predicts the rare classes scores 0.97 (traced above), and even on the balanced set above, 0.829 accuracy sat beside 0.457 minority recall.
• Duplicating minority images offline and then putting copies of the same image in both train and validation (leakage). Split first, then oversample.
• Using augmentations that change the class, or using the same strong augmentation on majority classes where it is unnecessary.
• Correcting the prior twice (weighted loss and logit adjustment at full strength) and then wondering why the majority classes suffer.
• Fine-tuning the whole network at the full learning rate on a few images, which erases the pretrained features.

Direct answer

I would handle it as four steps: find out whether the model errors are label errors, rank the suspect labels so reviewers spend time where wrong labels are densest, fix the labels and the guideline that produced them, and measure the effect on a test set whose labels were also audited. The sentence to say first in the room: the evidence for a label problem is the model's disagreement with the label that a human reviewer then sides with the model on, and the proof of the fix is a metric measured on clean evaluation labels.

Terms. A label is the class (or box, or mask) a human attached to an image. Label noise is the share of those that are wrong. Out-of-fold predictions come from a model that never saw the image it predicts: split the data into folds, train on all but one, predict the held-out one, rotate. Consensus means several annotators label the same item and a rule (majority, or a senior adjudicator) settles it.

Step 1: identify the problem (is it labels or is it the model?)

1. Look at the errors, not the aggregate. Pull a random sample of the model's confident mistakes (say 100) and have two people classify each: model wrong, label wrong, or ambiguous image. If a large fraction is label wrong, the metric is partly measuring annotation quality.
2. Look for structure. A confusion matrix (a table with true classes as rows and predicted classes as columns, so each off-diagonal cell counts one kind of mix-up) by class pair, a per-annotator error rate, a per-batch or per-date error rate. Noise concentrated in one class pair (two visually close classes) points to an ambiguous guideline; noise concentrated in one annotator or one week points to a process fault; scattered noise points to fatigue.
3. Check training behaviour: a model that fits training labels almost perfectly while validation lags is memorizing noise (fitting the wrong labels one by one instead of learning the pattern). Per-sample loss late in training is a ranking of suspects.
4. Compare with disagreement among annotators on an overlapped sample: it gives a rough ceiling on how clean the labels can be.

Step 2: prioritize

Rank by expected metric impact per hour of review: the suspect list from out-of-fold probabilities (lowest probability for the given label first), weighted toward the classes the product cares about, and with systematic problems (a guideline gap) before random ones (a single typo). Fix the cause at the same time as the symptoms; relabeling 5,000 images and leaving the ambiguous guideline in place recreates the problem.

Step 3 and a measured example: improve labels

The program makes 20% of the labels in a 1,100-image training set wrong on purpose (the 8x8 digit images that ship with scikit-learn, a stand-in for an image task), computes out-of-fold probabilities with a logistic regression, sends the 200 lowest-probability rows to a simulated reviewer who writes the true label, and compares the effect with spending the same 200 reviews on random rows. The final model is a small neural network trained until it fits the (noisy) training labels (its optimiser, lbfgs, is just scikit-learn's routine for that; it plays no part in the lesson); test labels are clean here, which real projects must arrange by auditing the test set. Ten seeds. A second run at the end changes only how the wrong labels are chosen: instead of a uniformly random other class, each wrong label is the look-alike class of the true one (0 with 6, 1 with 7, 2 with 3, 4 with 9, 5 with 8), which is closer to how real annotation errors tend to look.

import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[k] = "1"      # single-threaded linear algebra: faster here and repeatable; set before importing numpy
import warnings
import numpy as np
warnings.filterwarnings("ignore")   # lbfgs stops at its iteration cap on some fits; that is fine for this comparison
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import cross_val_predict, StratifiedKFold

X, y_true = load_digits(return_X_y=True); X = X / 16.0

PARTNER = np.array([6, 7, 3, 2, 9, 8, 0, 1, 5, 4])               # confusable pairs: 0-6, 1-7, 2-3, 4-9, 5-8
def experiment(seed, noise=0.2, budget=200, structured=False):
    r = np.random.default_rng(seed)
    idx = r.permutation(len(y_true)); tr, te = idx[:1100], idx[1100:]
    y = y_true[tr].copy()
    flip = r.choice(len(tr), int(noise * len(tr)), replace=False)
    if structured:
        y[flip] = PARTNER[y[flip]]                                     # always the look-alike class
    else:
        y[flip] = (y[flip] + r.integers(1, 10, len(flip))) % 10      # a different class, uniformly
    def fit_score(labels, rows=slice(None)):
        m = MLPClassifier((64,), solver="lbfgs", max_iter=300, random_state=seed).fit(X[tr][rows], labels)
        return m.score(X[te], y_true[te])
    base = fit_score(y)
    # suspect score: out-of-fold probability the model gives to the label the annotators wrote
    oof = cross_val_predict(LogisticRegression(max_iter=1000), X[tr], y, cv=StratifiedKFold(5, shuffle=True, random_state=seed),
                            method="predict_proba")
    p_given = oof[np.arange(len(tr)), y]
    suspects = np.argsort(p_given)[:budget]
    precision = float(np.mean(y[suspects] != y_true[tr][suspects]))
    y_fix = y.copy(); y_fix[suspects] = y_true[tr][suspects]         # a reviewer relabels the suspects
    fixed = fit_score(y_fix)
    rnd = r.choice(len(tr), budget, replace=False)
    y_rnd = y.copy(); y_rnd[rnd] = y_true[tr][rnd]                   # same review budget spent on random rows
    rand_fixed = fit_score(y_rnd)
    clean = fit_score(y_true[tr])
    return base, fixed, rand_fixed, clean, precision, float(np.mean(y != y_true[tr]))

rows = np.array([experiment(s) for s in range(10)])
names = ["noisy labels", "relabel top-200 suspects", "relabel 200 random rows", "all labels clean (ceiling)"]
print("training set 1100 images, 20% of labels wrong, review budget 200 rows, 10 seeds")
for n, col in zip(names, rows[:, :4].T):
    print(f"  {n:28s} test accuracy {col.mean():.3f} (std {col.std():.3f})")
print(f"  share of the 200 suspects that really were wrong: {rows[:, 4].mean():.2f}")
print(f"  actual share of wrong labels in the training set: {rows[:, 5].mean():.2f}")

# ---- the same experiment when the wrong labels are class-pair structured (a wrong label is always the look-alike class)
rows2 = np.array([experiment(s, structured=True) for s in range(10)])
print("same, but every wrong label is the look-alike class of the true one")
for n, col in zip(names, rows2[:, :4].T):
    print(f"  {n:28s} test accuracy {col.mean():.3f} (std {col.std():.3f})")
print(f"  share of the 200 suspects that really were wrong: {rows2[:, 4].mean():.2f}")

Output:

training set 1100 images, 20% of labels wrong, review budget 200 rows, 10 seeds
  noisy labels                 test accuracy 0.727 (std 0.019)
  relabel top-200 suspects     test accuracy 0.927 (std 0.010)
  relabel 200 random rows      test accuracy 0.767 (std 0.022)
  all labels clean (ceiling)   test accuracy 0.958 (std 0.008)
  share of the 200 suspects that really were wrong: 0.97
  actual share of wrong labels in the training set: 0.20
same, but every wrong label is the look-alike class of the true one
  noisy labels                 test accuracy 0.736 (std 0.014)
  relabel top-200 suspects     test accuracy 0.889 (std 0.016)
  relabel 200 random rows      test accuracy 0.782 (std 0.009)
  all labels clean (ceiling)   test accuracy 0.958 (std 0.008)
  share of the 200 suspects that really were wrong: 0.81

Reading the first block. Of the 200 suspects, 0.97 were truly wrong, against a base rate of 0.20 wrong in the data, so the ranking concentrates reviewer effort almost five times better than random review. Spending the same 200 reviews on random rows lifts test accuracy from 0.727 to 0.767; spending them on the suspects lifts it to 0.927, close to the 0.958 ceiling from fully clean labels. This model memorizes its training labels, which makes it sensitive to noise; a more regularized model (one whose capacity to fit is restrained, for example by a penalty on large weights) would lose less to noise and gain less from the fix. The test labels are clean, which a real project must arrange by auditing them.

Reading the second block, which is the more realistic one. When every wrong label is the look-alike class, the model finds it harder to say the label is wrong, because the label it was given is a plausible answer for the image. The suspect list is now 0.81 wrong instead of 0.97, still about four times the 0.20 base rate, and relabeling it lifts test accuracy from 0.736 to 0.889 against 0.782 for random review. The ranking still beats random review by a wide margin, but about one suspect in five is a hard-but-correct example, which is why reviewers decide and the list is never applied automatically. Real noise is also not always symmetric, so on your data the share will differ again.

Other fixes in the same toolbox: overlap labeling and consensus for the classes with the highest disagreement; guideline changes with examples; removing items reviewers mark ambiguous rather than forcing a class; and noise-robust training as a supplement, not a substitute for cleaning. Two examples of noise-robust training: label smoothing (training against targets such as 0.9 for the given class and a small share spread over the others, instead of exactly 1, so one wrong label pulls the model less hard) and losses that down-weight large-loss samples (examples the model finds very hard, which are disproportionately the mislabeled ones, count for less in the update).

Step 4: measure the impact

• Audit the evaluation set first. If the test labels are noisy, the metric understates the model and the change in the metric after cleaning only the training set is hard to read. Re-adjudicate the test set (all of it if small; if large, a stratified sample, which takes the same share from every class, with a confidence interval around the corrected metric) and keep it frozen.
• Report before and after on the same audited test set, the per-class change, and the seed-to-seed spread (several training runs). Say what a reviewer-hour bought: the metric gain divided by review hours, against the random-review baseline above.
• Watch the metric you care about in production, not only the offline one: a false negative rate on the defect class, for example.

How I would tell it as a story

The question asks for an experience, so the answer has to be sayable. This is the structure, with square brackets where your own facts and numbers go; do not borrow numbers from the program above, which runs on a stand-in dataset.

Situation: a vision model plateaued and its errors looked odd. Task: find out if labels were the cause. Action: sampled confident errors, found the label-wrong share was large and concentrated in two classes, ranked suspects out-of-fold, had reviewers relabel the top of the list, rewrote the guideline for the two classes with examples, and audited the test set. Result: report the audited-test-set change, the per-class change, and what remained noisy.

As it might be said aloud, in about a minute:

On [project], our [image classifier] had plateaued at [metric], and some of its confident mistakes
looked like the model was right and the label was wrong. Before touching the model, I pulled about
a hundred of those confident mistakes and two of us sorted each one: model wrong, label wrong, or
ambiguous. [X percent] were label wrong, and most sat in [two class pairs], so I treated it as a
guideline problem and not just typos. I ranked the training images by how little probability a
model that had never seen them gave to their label, and had reviewers relabel from the top of that
list. In parallel I rewrote the guideline for those classes with example images. I also had the
test set re-adjudicated, because without that I could not trust any before and after number. On
the audited test set, [metric] went from [a] to [b], mostly in [those classes]; [what stayed noisy,
and what I would do next].

If you have no such project, say so and present the same sequence as how you would do it; the method is what is being tested.

Mistakes

• Cleaning only the training labels and judging on the noisy test set.
• Dropping every low-probability example. The low-probability examples include hard but correct ones, so the remaining training set contains mostly cases the model already finds easy; a model trained only on easy cases never sees the hard ones, then assigns high probability (becomes overconfident, meaning its stated probabilities exceed how often it is right) to hard cases it gets wrong.
• Fixing symptoms and not the guideline or the annotator process.
• Claiming improvement from one training run.