Computer Vision Questions

Direct answer

Both methods automate the choice of image augmentations, and they differ in how much they search. AutoAugment searches a large space of policies for the one with the best validation accuracy on the target dataset. RandAugment removes that search: it applies randomly chosen operations from a fixed list and leaves only a few knobs, in the common formulation the number of operations per image (N) and one shared strength (M), which you tune with a small grid on the real task. So AutoAugment costs far more compute and has far more to tune but can express a tailored policy; RandAugment is cheap and simple but is only as good as its operation list. The published results favour the cheap method: the RandAugment paper reports matching or surpassing earlier automated augmentation approaches, AutoAugment included, on CIFAR-10/100, SVHN and ImageNet, and coming within 0.3% mAP of AutoAugment on COCO detection, without a separate search phase. AutoAugment's own paper reports a CIFAR-10 error of 1.5% and ImageNet top-1 accuracy of 83.5%, which were records when published, so searched policies can beat the standard hand-written augmentation on large benchmarks. Whether they beat a manual choice on your data is a question for your own validation set, and a manual policy that already encodes the invariances your deployment needs is the baseline to beat (an invariance is a change to the input that should not change the answer, such as a digit shifted by a pixel or a photo taken in dimmer light). Automatic policies are most useful when you do not know them, when data is limited, and when the model is large enough to overfit; the toy below shows the mechanics of choosing by validation accuracy, and it does not reproduce the benchmark results above.

Terms. Augmentation creates modified copies of training images (shifted, rotated, blurred, recoloured) so the model learns to ignore changes that should not alter the label. An operation is one such transform with a magnitude (how strong). A policy is a recipe of operations with probabilities and magnitudes. Validation accuracy is accuracy on held-out labeled data used to choose settings. The two methods are described in the original papers (AutoAugment, 2018; RandAugment, 2019): the first uses a search algorithm over policies made of sub-policies of two operations, each with a probability and a magnitude, scored by validation accuracy; the second needs no separate search or proxy task. A proxy task is a cheaper stand-in for the real training job (a smaller network, fewer epochs or a subset of the data) used to score each candidate policy during the search; a policy that is best on the proxy may not be best on the real task. A sub-policy is one pair of operations applied in sequence, each with its own probability of being applied and its own magnitude; a policy is a set of sub-policies, and each image gets one of them at random. Regularization is anything that stops a model fitting the quirks of its training set, and augmentation is a regularizer because the model never sees exactly the same image twice. A model overfits when its accuracy on training images is far above its accuracy on new ones. The grid of N and M in RandAugment is simply every combination of the few values you try (N in {1, 2} and M in {0.25, 0.5, 1.0} gives 2 x 3 = 6 combinations, each needing one training run).

Comparison

A measurement on a toy task

This is a stand-in for an image model: logistic regression on scikit-learn's 8 by 8 digit images, trained on 600 images. It trains the model on unshifted, undimmed images and tests on images altered the way a deployment might alter them, in two scenarios. Scenario A: every validation and test image is shifted by up to one pixel in each direction, and the team's manual policy (shift only) is the right guess. Scenario B: every validation and test image is dimmer and noisier (a poor sensor), and the team's manual guess is still "shift only", which is now the wrong guess. The four operations are shift, noise, vertical flip (which makes 6 and 9, among others, harder to tell apart) and brightness. Policies compared: none; the manual shift-only policy; a RandAugment-style grid over N in {1, 2} and magnitude in {0.25, 0.5, 1.0} (6 runs), once with all four operations and once with the flip removed; and a random search over 10 policies of three (operation, probability, magnitude) triples. The toy is deliberately small and its operation list contains a harmful operation, so its ranking (manual best in scenario A, no augmentation as good as anything in scenario B) is a property of this setup and not evidence about the real methods. That random search is only a stand-in for AutoAugment: the real method searches a vastly larger space with a learning-based search algorithm, so these numbers describe this toy and say nothing about how well the real methods perform. In each scenario the validation images are altered like the test images, each configuration is chosen by validation accuracy and scored on the test set, and results are averaged over 5 seeds (each seed redraws the split).

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression

X0, y0 = load_digits(return_X_y=True); X0 = X0.reshape(-1, 8, 8) / 16.0
OPS = ["shift", "noise", "flipud", "brightness"]

def shift_img(img, dx, dy):
    out = np.zeros_like(img)
    ys, xs = slice(max(dy, 0), 8 + min(dy, 0)), slice(max(dx, 0), 8 + min(dx, 0))
    yd, xd = slice(max(-dy, 0), 8 + min(-dy, 0)), slice(max(-dx, 0), 8 + min(-dx, 0))
    out[ys, xs] = img[yd, xd]
    return out

def apply(img, op, m, rng):                      # m in [0, 1] is the magnitude
    if op == "shift":
        k = int(round(2 * m)); return shift_img(img, int(rng.integers(-k, k + 1)), int(rng.integers(-k, k + 1)))
    if op == "noise": return np.clip(img + rng.normal(0, 0.3 * m, img.shape), 0, 1)
    if op == "flipud": return img[::-1]          # ignores magnitude; changes which digit looks like which
    return np.clip(img * (1 + rng.uniform(-0.5, 0.5) * m), 0, 1)

def augment(X, y, policy, rng, copies=2):
    """policy = ("rand", ops, N, M): N ops drawn uniformly from ops, one shared magnitude M (RandAugment style)
       policy = ("list", [(op, prob, magnitude), ...]): every listed op tried independently (policy-search style)"""
    Xs, ys = [X], [y]
    for _ in range(copies):
        new = []
        for img in X:
            if policy[0] == "rand":
                _, ops, n, m = policy
                for op in rng.choice(ops, size=n): img = apply(img, str(op), m, rng)
            else:
                for op, p, m in policy[1]:
                    if rng.random() < p: img = apply(img, op, m, rng)
            new.append(img)
        Xs.append(np.array(new)); ys.append(y)
    return np.concatenate(Xs).reshape(-1, 64), np.concatenate(ys)

def fit_score(policy, data, rng):
    (Xtr, ytr), (Xev, yev) = data
    Xa, ya = augment(Xtr, ytr, policy, rng) if policy else (Xtr.reshape(-1, 64), ytr)
    return LogisticRegression(max_iter=200).fit(Xa, ya).score(Xev.reshape(-1, 64), yev)

def shifted(X, rng):                             # deployment A: every image moved by up to 1 pixel
    return np.array([shift_img(i, int(rng.integers(-1, 2)), int(rng.integers(-1, 2))) for i in X])

def dim_noisy(X, rng):                           # deployment B: every image dimmer and noisier (a poor sensor)
    return np.clip(X * rng.uniform(0.5, 1.0, (len(X), 1, 1)) + rng.normal(0, 0.25, X.shape), 0, 1)

def one_seed(seed, deploy, manual):
    rng = np.random.default_rng(seed); idx = rng.permutation(len(y0))
    tr, va, te = idx[:600], idx[600:900], idx[900:]
    Xtr, ytr = X0[tr], y0[tr]
    Xva, Xte = deploy(X0[va], rng), deploy(X0[te], rng)       # validation mirrors deployment
    val, test = ((Xtr, ytr), (Xva, y0[va])), ((Xtr, ytr), (Xte, y0[te]))
    def choose(cands):                                         # pick by validation accuracy, report test accuracy
        v = [fit_score(p, val, np.random.default_rng(seed)) for p in cands]
        return cands[int(np.argmax(v))]
    sc = lambda pol, d=test: fit_score(pol, d, np.random.default_rng(seed))
    res = {"none": sc(None), "manual guess: shift only": sc(manual)}
    for name, ops in (("RandAugment grid, all 4 ops", OPS), ("RandAugment grid, flipud removed", [o for o in OPS if o != "flipud"])):
        grid = [("rand", ops, n, m) for n in (1, 2) for m in (0.25, 0.5, 1.0)]           # 6 training runs
        res[name] = sc(choose(grid))
    pols = [("list", [(OPS[rng.integers(4)], float(rng.choice([0.2, 0.5, 0.8, 1.0])), float(rng.choice([0.25, 0.5, 0.75, 1.0]))) for _ in range(3)]) for _ in range(10)]
    res["random policy search, 10 policies"] = sc(choose(pols))
    return res

SHIFT_ONLY = ("list", [("shift", 1.0, 0.5)])
for title, deploy in (("A: deployment moves the digit by up to 1 pixel (the team's manual guess is right)", shifted),
                      ("B: deployment is dimmer and noisier (the team's manual guess, shift only, is wrong)", dim_noisy)):
    R = [one_seed(s, deploy, SHIFT_ONLY) for s in range(5)]
    print("Scenario " + title)
    print("policy                               test accuracy, mean +- sd over 5 seeds")
    for k in R[0]:
        v = np.array([r[k] for r in R]); print(f"{k:36s} {v.mean():.3f} +- {v.std():.3f}")
print("training runs spent on the search: RandAugment grid 6, random policy search 10")
print("grid configurations here:", 2 * 3, "| random policies here:", (4 * 4 * 4) ** 3)
print("illustrative AutoAugment-style space (16 ops, 11 probs, 10 magnitudes, 2 ops per sub-policy, 5 sub-policies):", f"{(16 * 11 * 10) ** 10:.1e}")

Output:

Scenario A: deployment moves the digit by up to 1 pixel (the team's manual guess is right)
policy                               test accuracy, mean +- sd over 5 seeds
none                                 0.363 +- 0.010
manual guess: shift only             0.585 +- 0.012
RandAugment grid, all 4 ops          0.403 +- 0.012
RandAugment grid, flipud removed     0.487 +- 0.009
random policy search, 10 policies    0.466 +- 0.052
Scenario B: deployment is dimmer and noisier (the team's manual guess, shift only, is wrong)
policy                               test accuracy, mean +- sd over 5 seeds
none                                 0.794 +- 0.016
manual guess: shift only             0.581 +- 0.006
RandAugment grid, all 4 ops          0.693 +- 0.021
RandAugment grid, flipud removed     0.773 +- 0.015
random policy search, 10 policies    0.806 +- 0.015
training runs spent on the search: RandAugment grid 6, random policy search 10
grid configurations here: 6 | random policies here: 262144
illustrative AutoAugment-style space (16 ops, 11 probs, 10 magnitudes, 2 ops per sub-policy, 5 sub-policies): 2.9e+32

What it shows, on this toy only. In scenario A, augmentation that matches the deployment shift is worth more than any amount of search (0.585 manual against 0.363 with none). Uniformly sampling from an operation list that contains a harmful operation gives 0.403, and removing that one operation raises it to 0.487, so the list matters as much as the tuning; a real RandAugment list is built from operations known to be safe for the data, and the flip is here on purpose as the bad entry. In scenario B the same manual shift-only guess is wrong and costs accuracy: 0.581 against 0.794 with no augmentation at all, a 0.213 loss from a confident, unverified choice. The validation-driven methods avoid that loss: the flip-free grid reaches 0.773 and the random search 0.806, which is about the same as no augmentation (0.794; the standard deviations are 0.015 to 0.016), so in this toy there was little for a search to gain over doing nothing in scenario B. No searched policy clearly beat doing nothing there, and none beat the right manual guess in scenario A. What the validation set bought was protection from a bad choice, and the cheapest way to get that is also the simplest: score the manual policy against no augmentation on validation images (2 runs) before spending 6 or 10 more. The random policy search varies the most across seeds in scenario A (standard deviation 0.052 against 0.009 to 0.012 for the others), because 10 random policies scored on 300 validation images sometimes pick a poor one. The search signal in both scenarios is the validation set, which only reflects what the validation images contain.

Search-space sizes, multiplied out. The grid has 2 x 3 = 6 configurations. The random policy space is $(4 \times 4 \times 4)^3 = 64^3 = 262{,}144$: each of 3 triples picks one of 4 operations, 4 probabilities and 4 magnitudes. For the illustrative AutoAugment-style setting, one operation slot has $16 \times 11 \times 10 = 1{,}760$ choices (16 operations, 11 probabilities, 10 magnitudes); a sub-policy has 2 such slots and a policy has 5 sub-policies, so 10 slots in all, giving $1760^{10} \approx 2.9 \times 10^{32}$. That number comes from this arithmetic, not from a statement about any published setting.

When do searched policies beat manual augmentation?

• When you do not know which invariances matter (unfamiliar imagery, many classes) and a large benchmark-style dataset and model are available, as in the published CIFAR-10, ImageNet and SVHN results above. Scenario B is a small reminder that a confident manual guess can be wrong, and that validating it against no augmentation catches that.
• When the training set is small and the model large, so regularization from varied copies is the main lever.
• When you can afford the search or can reuse a policy found on a similar dataset (check it transfers: validate it on yours).
• They do not help when the augmentations are mismatched with deployment (the flip case above), when validation data does not represent deployment, or when the weakness is something augmentation cannot create, such as a missing class of object.

Trade-offs and mistakes

Tuning strength on a validation set too small to separate the options; copying a policy tuned on natural photos to medical or document images, where a flip or colour shift can change the label; counting only the search cost and forgetting that stronger augmentation also needs longer training to converge; comparing a searched policy to no augmentation instead of to a sensible manual baseline, and, in the other direction, crediting a search for avoiding a bad manual choice when no augmentation would have avoided it equally well.

The idea in plain words

A pretrained backbone is the body of a network (the stack of layers that turns an image into a feature vector) that was already trained on a large image collection. Transfer learning reuses it, because the early layers learn generic things (edges, textures, simple shapes) that almost any image task needs. There are two ways to reuse it:

• Feature extraction: freeze the backbone (its weights never change) and train only a new final layer, the head, that maps the feature vector to your classes. The backbone is a fixed function; you learn a small classifier on top of its outputs.
• Full fine-tuning: start from the pretrained weights, replace the head, and let all layers keep learning, with a small learning rate so the useful pretrained features are adjusted rather than destroyed.

Feature extraction has few trainable parameters, so it is cheap, stable and hard to overfit; fine-tuning has many, so it can adapt the features to your images but can overfit a small dataset and forget what it started with.

Side by side

A middle path is common: train the head first, then unfreeze only the last blocks (the layers closest to the output, which are the most task-specific), keeping the early generic layers frozen.

What each strategy is allowed to change, with counted numbers

No accuracy race between the strategies is shown here, because no pretrained ResNet is available to the program. A race on a small stand-in network pretrained on a few hundred tiny digit images would say little about a real backbone pretrained on a large, diverse image collection, and in such a toy training from scratch can tie the transfer methods, which is not what is seen in practice. What is widely observed in practice, and the reason to start with feature extraction, is that with a few hundred to a few thousand labelled images a pretrained backbone usually beats training from scratch by a clear margin. The advantage shrinks as the target dataset and the training time grow: a published study of detection and segmentation (He, Girshick and Dollar, 2018) found that pretraining mainly speeds up early convergence and does not necessarily improve final accuracy once the target data is large, which is why the from-scratch comparison in step 7 is only worth running when you have the budget. What the program below does show is the counted reason (how many numbers each strategy has to fit from your labels) and one trap (BatchNorm: a layer that normalises activations using the mean and spread of the current batch during training, and keeps running averages of them for use at inference). The network is a tiny stand-in with the same structure as a real one: a backbone ending in a feature vector, plus a head.

import torch
import torch.nn as nn
from sklearn.datasets import load_digits

torch.set_num_threads(1)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32).reshape(-1, 1, 8, 8)

class Net(nn.Module):
    def __init__(self, n_out):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(),
            nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.AdaptiveAvgPool2d(2), nn.Flatten())            # 32 * 2 * 2 = 128 features
        self.head = nn.Linear(128, n_out)                     # the new final layer: 128 -> 2 classes
    def forward(self, x):
        return self.head(self.backbone(x))

def n_trainable(m):
    return sum(p.numel() for p in m.parameters() if p.requires_grad)

# 1. Count what each strategy is allowed to change, and what that means per labelled image.
model = Net(2)
all_params = n_trainable(model)
model.backbone.requires_grad_(False)                          # feature extraction: freeze the backbone
head_params = n_trainable(model)
print(f"trainable parameters: head only = {head_params}, all layers = {all_params}")
for n_labels in (20, 2000):
    print(f"  with {n_labels:>4} labelled images: head only = {head_params / n_labels:6.1f} parameters per image, all layers = {all_params / n_labels:6.1f}")

# 2. BatchNorm: frozen weights do not freeze the running statistics unless the layer is in eval mode.
def bn_drift(put_in_eval):
    torch.manual_seed(0)
    net = Net(2)
    net.backbone.requires_grad_(False)
    before = net.backbone[1].running_mean.clone()
    net.train()
    if put_in_eval:
        net.backbone.eval()
    net(X[:64] * 3.0 + 1.0)                                   # a batch whose statistics differ from the default
    return (net.backbone[1].running_mean - before).abs().max().item()

print(f"BatchNorm running_mean change with frozen weights, backbone in train mode: {bn_drift(False):.4f}")
print(f"BatchNorm running_mean change with frozen weights, backbone in eval mode : {bn_drift(True):.4f}")

Output:

trainable parameters: head only = 258, all layers = 5154
  with   20 labelled images: head only =   12.9 parameters per image, all layers =  257.7
  with 2000 labelled images: head only =    0.1 parameters per image, all layers =    2.6
BatchNorm running_mean change with frozen weights, backbone in train mode: 0.2160
BatchNorm running_mean change with frozen weights, backbone in eval mode : 0.0000

Reading the output:

• Parameters: a frozen backbone leaves 258 trainable parameters (128 features times 2 classes, plus 2 biases) against 5154 when everything trains. With 20 labelled images that is about 13 parameters per image for the head against about 258 per image for full fine-tuning; with 2000 images it is 0.1 against 2.6. Many parameters per labelled image is the situation in which a model can memorise the training set, which is why feature extraction is the safe start on tiny data and fine-tuning becomes reasonable as labels grow.
• BatchNorm: with every weight frozen, the layer's running mean still moved by 0.2160 when the backbone ran in training mode, and by 0.0000 in evaluation mode. Training a head on a small, differently distributed dataset with BatchNorm in training mode silently rewrites the backbone's statistics. Put the frozen backbone in evaluation mode.
• Learning rates, with example values: the learning rate is the size of the step the optimiser takes along the gradient on each update. A new head starts random and must move far, so give it a normal rate such as 1e-3; the pretrained backbone is already good and only needs nudging, so give it 10 to 100 times less, such as 1e-4 to 1e-5. These are starting values to validate on your data, not measured optima.

Applying a pretrained ResNet to a small labelled dataset: the steps

A ResNet is a family of CNNs built from residual blocks (blocks whose input is added back to their output, which makes deep networks trainable). The steps do not depend on the depth.

1. Use the weights' own preprocessing: the input size, channel order and per-channel mean and standard deviation the weights were trained with. A mismatch here silently lowers accuracy and raises no error.
2. Replace the head: swap the final fully connected layer for a new one with one output per class (read the backbone's feature width from the layer itself rather than assuming it).
3. Baseline first (feature extraction): freeze the backbone, put it in evaluation mode, train the head. This takes minutes, and its validation score is the number to beat.
4. Fine-tune if the baseline leaves room: unfreeze the last block or two, then more if data allows. Give the head a normal learning rate and the unfrozen backbone a learning rate roughly 10 to 100 times smaller (a starting range to validate, not a rule). A short warm-up (start with a tiny learning rate and ramp up over the first steps so the random head does not push big gradients into the backbone) and a decaying schedule (lower the rate gradually as training goes on) help.
5. Augment and regularise: flips and crops that suit the data, then regularisers (techniques that stop the model memorising): weight decay (a small penalty that pulls weights toward zero), label smoothing (training against softened targets, for example 0.9 on the correct class with the remaining 0.1 spread over the other classes, instead of a hard 1 and 0, so the model is not pushed to be over-confident), or dropout (randomly zeroing some activations during training) in the head; stop on validation loss. On a small dataset the augmentation and the validation split matter most; treat the regularisers as extras to add one at a time if validation shows overfitting.
6. Validate honestly: a stratified split (each class keeps its share), cross-validation (repeating the train and validate cycle on several different splits and averaging) when the set is tiny, and check that near-duplicate images do not sit on both sides of the split.
7. Compare against training from scratch only if you have the budget; with a few hundred images it rarely wins.

Trade-offs and mistakes

• Fine-tuning everything at the head's learning rate wrecks the pretrained features in the first few steps.
• Forgetting the BatchNorm mode, as measured above.
• Reporting accuracy from a split where near-duplicates leak across train and validation.
• Choosing by training accuracy instead of validation.

Direct answer

Treat synthetic data as a cheap source of exact labels for the rare classes, and real human-labeled data as the judge. The pipeline has four parts: (1) a small, carefully labeled REAL validation and test set that synthetic data never touches, with every rare class represented; (2) a synthetic generator whose labels are exact by construction (paste real object cutouts, or render, with randomised appearance); (3) a gap check run before training, which asks whether a simple classifier can tell synthetic from real; and (4) a human-in-the-loop labeling stream for real images, focused on the rare classes and using the model's own predictions as a first draft for people to correct. The only number that says the synthetic data helps is performance on the held-out real set.

Terms. Instance segmentation: find each object and outline its exact pixels, one mask per object. Synthetic-to-real gap (domain gap): the differences in look between generated and real images (background texture, noise, blur, lighting, object appearance) that a model can latch onto instead of the object. Domain randomization: vary those nuisance factors widely so the model cannot rely on any one of them. Human-in-the-loop: people correct or confirm machine output instead of labeling from scratch.

Pipeline, in the order I would build it

1. Fix the evaluation first. Collect a real set for validation and testing, stratified (sampled so that every class, including each rare one, is guaranteed a share of the set) so each rare class has enough examples to estimate its own average precision, or AP (for one class, the area under its precision-recall curve: how well true detections are ranked above false ones). A handful of examples gives a noisy number, so say how few you have. Keep synthetic images, and real images that are near-duplicates of training images (the same scene, slightly cropped or shifted), out of it.
2. Make the synthetic generator. Two common routes. (a) Copy-paste: cut real objects out using their masks, paste them on varied real backgrounds at random scale, rotation and position; the pasted mask IS the label. (b) Rendering or simulation of 3D models with automatic labels, useful when real cutouts of the rare class barely exist. Either way, randomise the nuisance factors: background, lighting, blur, noise level, occlusion, scale, and camera artifacts.
3. Handle occlusion honestly. A later paste hides part of an earlier one, so the label for the earlier object must be its VISIBLE pixels. The last block of the program pastes three objects of 87, 88 and 87 pixels. Object 1 is covered by 60 pixels of object 2, which was pasted after it, leaving 28 visible. Object 0 is covered by objects 1 and 2 together: 43 pixels are hidden (the two later objects overlap each other inside object 0, so the hidden regions are not simply added), leaving 44 visible. Object 2 was pasted last, so nothing hides it and all 87 pixels stay. The label of each object is its visible pixel set, not its full pasted shape.
4. Measure the gap before training. Train a small classifier (a domain classifier) to separate synthetic images from real ones using cheap whole-image statistics, such as mean brightness, the spread of brightness, and how sharp or noisy the edges are. Score it with AUC (area under the ROC curve): the chance that it gives a random synthetic image a higher synthetic-score than a random real one, so 0.5 means it cannot tell them apart and 1.0 means it separates them perfectly. A score near 1.0 is a warning that the generated images carry cues, such as hard edges or a flat background, that a detector COULD use as a shortcut; it does not by itself prove the detector will, and it does not tell you how much the shortcut will cost. There is no universal passing value, so use the number as a trend: it should fall as you improve the generator, and the final judge stays the accuracy or AP on the held-out real set. This is a diagnostic, not a guarantee: a low score means the cheap statistics agree, not that the images match in every way.
5. Train on a mix, then fine-tune. Pretrain or co-train on synthetic plus real, then fine-tune on real only, or down-weight synthetic samples. Choose the mix ratio on the real validation set, not by feel.
6. Human-in-the-loop labeling for the rare classes. Run the current model on unlabeled real images and send to annotators the images with (a) the highest rare-class scores, to find more positives, and (b) the most uncertain predictions. Annotators correct a drafted mask instead of drawing one, which is faster than labeling from scratch; this model-assisted labeling is the human half of the hybrid, and the synthetic generator is the other half. Check label quality by having two annotators label a sample and computing mask IoU between them. Retrain, re-measure on the real validation set, repeat.
7. Report gap evidence together with the result. Per-class AP on real data with and without synthetic data, the domain-classifier score, and a visual spot check of failures.

Runnable version

This is a small stand-in world, not a real detector. "Real" images are generated with a textured background, sensor noise, soft edges and shading. The rare class is an elongated ellipse, the common class a rounder one. "raw" synthetic images are naive pastes on a flat background with hard edges; "rand" synthetic images paste the same objects with randomised background texture, noise, blur and shading. Because the "real" generator is itself part of this stand-in, the run shows how the diagnostics behave, not how much real data you need. Part 1 is the domain classifier: a logistic regression (a linear model that outputs a probability) on six global statistics (mean brightness, brightness spread, the spread and the maximum of the Laplacian, which responds to sharp edges and noise, and the maximum and median edge-gradient strength), scored by 5-fold cross-validated AUC (five train/test splits of the data, averaging AUC on the held-out part of each), over 5 seeds. Part 2 trains a classifier for the rare class on 15 real images plus optional synthetic images and tests on 600 real images, over 10 seeds. Part 2 reads a different feature, the shape of the bright region (how elongated it is), not the texture statistics of Part 1, so the two parts are separate experiments and Part 1's AUC does not predict Part 2's accuracy. Part 3 is the occlusion example.

import numpy as np
from scipy.ndimage import gaussian_filter, laplace
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

H = 32
yy, xx = np.mgrid[0:H, 0:H]

def ellipse_mask(cx, cy, a, b, theta):
    c, s = np.cos(theta), np.sin(theta)
    u = (xx - cx) * c + (yy - cy) * s; v = -(xx - cx) * s + (yy - cy) * c
    return (u / a) ** 2 + (v / b) ** 2 <= 1.0

def draw(g, rare, mode):
    """mode 'real' = textured background + sensor noise + soft edges + shading (our stand-in for real photos);
    'raw' = naive paste (flat background, hard edges, no noise); 'rand' = the same paste with randomised realism."""
    a, b = (9.0, 3.0) if rare else (6.0, 5.0)                       # rare class is elongated, common class is round
    m = ellipse_mask(g.uniform(10, 22), g.uniform(10, 22), a * g.uniform(.8, 1.2), b * g.uniform(.8, 1.2), g.uniform(0, np.pi))
    if mode == "real":
        bg = 0.5 + 0.3 * gaussian_filter(g.normal(size=(H, H)), 4) * 4; noise = 0.03; blur = 1.0; shade = 0.15
    elif mode == "raw":
        bg = np.full((H, H), 0.5); noise = 0.0; blur = 0.0; shade = 0.0
    else:
        bg = 0.5 + g.uniform(0, 0.4) * gaussian_filter(g.normal(size=(H, H)), g.uniform(2, 6)) * 4
        noise = g.uniform(0, 0.06); blur = g.uniform(0.3, 1.5); shade = g.uniform(0, 0.2)
    obj = 0.85 - shade * (xx - xx.mean()) / H
    img = np.where(m, obj, bg)
    if blur > 0: img = gaussian_filter(img, blur)
    img = img + g.normal(0, noise, size=img.shape) if noise > 0 else img
    return img.astype(np.float32), m

def features(img):                                                   # cheap global statistics
    lap = laplace(img); gy, gx = np.gradient(img); gm = np.hypot(gx, gy)
    return [img.mean(), img.std(), lap.std(), np.abs(lap).max(), gm.max(), np.percentile(gm, 50)]

def domain_auc(mode, n=300, seed=0):
    g = np.random.default_rng(seed)
    X = [features(draw(g, g.random() < .5, "real")[0]) for _ in range(n)] + [features(draw(g, g.random() < .5, mode)[0]) for _ in range(n)]
    y = [0] * n + [1] * n
    clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
    return cross_val_score(clf, np.array(X), np.array(y), cv=5, scoring="roc_auc").mean()

print("Domain-classifier AUC, real vs synthetic (0.5 = cannot tell them apart; 1.0 = trivially different):")
for mode in ("raw", "rand"):
    print(f"  {mode:5s} paste: {np.mean([domain_auc(mode, seed=s) for s in range(5)]):.3f}  (mean over 5 seeds)")

# --- Does adding synthetic data help a rare-class classifier evaluated on REAL images? ---
def pixel_feats(img):                                                # shape of the bright region: spread along its two axes
    w = np.clip(img - 0.65, 0, None) + 1e-6
    cx, cy = (w * xx).sum() / w.sum(), (w * yy).sum() / w.sum()
    cov = np.array([[(w * (xx - cx) ** 2).sum(), (w * (xx - cx) * (yy - cy)).sum()],
                    [(w * (xx - cx) * (yy - cy)).sum(), (w * (yy - cy) ** 2).sum()]]) / w.sum()
    ev = np.sort(np.linalg.eigvalsh(cov))
    return np.array([ev[1] / ev[0], np.log(ev[1]), np.log(ev[0]), w.sum() / 100])

def dataset(g, n, mode, rare_frac):
    ys = (g.random(n) < rare_frac).astype(int)
    return np.array([pixel_feats(draw(g, bool(y), mode)[0]) for y in ys]), ys

print("\nRare-class accuracy on 600 REAL test images (balanced); 15 real training images; mean over 10 seeds")
print("synthetic added | accuracy")
res = {}
for n_syn, mode in ((0, "raw"), (300, "raw"), (300, "rand")):
    accs = []
    for seed in range(10):
        g = np.random.default_rng(100 + seed)
        Xr, yr = dataset(g, 15, "real", .5); Xt, yt = dataset(np.random.default_rng(900 + seed), 600, "real", .5)
        if n_syn:
            Xs, ys = dataset(g, n_syn, mode, .5); Xr, yr = np.vstack([Xr, Xs]), np.concatenate([yr, ys])
        clf = make_pipeline(StandardScaler(), LogisticRegression(C=0.1, max_iter=2000)).fit(Xr, yr)
        accs.append(clf.score(Xt, yt))
    label = "none" if n_syn == 0 else f"{n_syn} {mode}"
    print(f"{label:>15s} | {np.mean(accs):.3f} (min {min(accs):.3f}, max {max(accs):.3f})")

# --- Exact labels by construction, with occlusion: later pastes hide earlier ones ---
g = np.random.default_rng(3)
canvas = np.full((H, H), 0.5); owner = -np.ones((H, H), dtype=int); masks = []
for k in range(3):
    m = ellipse_mask(g.uniform(10, 22), g.uniform(10, 22), 7, 4, g.uniform(0, np.pi)); masks.append(m)
    canvas[m] = 0.85; owner[m] = k                                   # a later paste overwrites the owner map
pasted = [int(m.sum()) for m in masks]
visible = [int((owner == k).sum()) for k in range(3)]
print("\npasted mask pixels:", pasted, "visible after occlusion:", visible)
for k in range(3):
    later = np.zeros((H, H), dtype=bool)
    for j in range(k + 1, 3): later |= masks[j]                      # union of every later paste
    print(f"object {k}: pasted {pasted[k]} - covered by later pastes {int((masks[k] & later).sum())} = visible {visible[k]}")

Output:

Domain-classifier AUC, real vs synthetic (0.5 = cannot tell them apart; 1.0 = trivially different):
  raw   paste: 1.000  (mean over 5 seeds)
  rand  paste: 0.760  (mean over 5 seeds)

Rare-class accuracy on 600 REAL test images (balanced); 15 real training images; mean over 10 seeds
synthetic added | accuracy
           none | 0.526 (min 0.457, max 0.662)
        300 raw | 0.626 (min 0.590, max 0.668)
       300 rand | 0.651 (min 0.622, max 0.677)

pasted mask pixels: [87, 88, 87] visible after occlusion: [44, 28, 87]
object 0: pasted 87 - covered by later pastes 43 = visible 44
object 1: pasted 88 - covered by later pastes 60 = visible 28
object 2: pasted 87 - covered by later pastes 0 = visible 87

Reading it. A naive paste is perfectly separable from real (AUC 1.000), and randomising the realism lowers that to 0.760, which still means the synthetic and real images differ in ways these statistics can detect. In part 2, adding 300 raw synthetic images to 15 real ones lifts mean accuracy on real test images from 0.526 to 0.626 and the randomised version gives 0.651. The difference between raw and randomised (0.025) is small compared with the spread over seeds (ranges of 0.59 to 0.67 and 0.62 to 0.68 overlap), so this run does not show that randomisation beats the naive paste.

What this toy does and does not show: it shows that with only 15 real images, extra labeled synthetic examples can add signal. It does not reproduce the real-world lesson of step 4, because the Part 2 classifier looks only at the shape of the bright region, a cue the raw paste gets right, so the texture shortcut that gives the raw paste an AUC of 1.000 never matters to it. A real detector looks at texture, edges and noise and can latch onto those cues, which is why the gap check, domain randomisation and the real-only validation set are standard practice. Treat the accuracy numbers as crude (the features are simple) and never as a measurement of how much randomisation or synthetic data would help a real model.

Trade-offs and failure modes

• The model can learn the synthetic shortcut: for example, "pasted objects have hard edges". Symptoms: validation on synthetic data near perfect, real data poor. Diagnosis: the domain classifier and a per-source error breakdown. Fix: randomise or match the artifacts, mix in real data, fine-tune on real.
• Label noise in the loop. Model-drafted masks bias annotators toward the model's errors (automation bias: people tend to accept a machine's suggestion even when it is wrong). Counter with spot audits where annotators label a few images with no draft, and measure agreement.
• Class imbalance in the loop. If you only fetch top-scoring images you may miss rare-class appearances the model cannot yet detect; also sample at random and from sources likely to hold the rare class.
• Cost. Rendering and compute are cheap per image; person-hours are not. Spend them on the real validation set, quality audits and the hard cases.
• Common mistakes. Evaluating on synthetic data, leaking cutouts from test images into the paste library, labeling pasted objects by their full outline when they are occluded, and adding unlimited synthetic data without checking the mix ratio.

The idea in plain words

Tiny objects fail in a detector for three separable reasons, and each has its own fix. First, the network's stride (how many input pixels one feature-map cell spans) can be larger than the object, so no cell describes it. Second, shrinking the picture to fit the GPU deletes the few pixels the object has. Third, the training signal is thin: a box a few pixels wide is hard to match by IoU (intersection over union, the overlap score between two boxes), and most random training crops contain nothing. Memory is the constraint that pushes you towards the wrong fix (downscale the image), so the plan is: keep native resolution, fit it into memory by cropping, and spend the extra compute only where small objects live.

The plan, in the order I would do it

1. Do not downscale the whole image; train and infer on crops (tiles) at native resolution. The program assumes 4096 pixel aerial images with objects 6 to 20 pixels wide. Resizing to 1024 would shrink the median object from 11.3 to 2.8 pixels and leave every object under 8 pixels; resizing to 2048 still leaves 0.80 of them under 8 pixels. Cropping keeps all 11.3.
2. Fix the stride before anything else. A 10-pixel object covers 2.5 cells per side at stride 4, 1.25 at stride 8, 0.62 at stride 16 and 0.31 at stride 32. Use a feature pyramid (maps at several strides, merged top-down so that fine maps also carry semantic features) whose finest level has stride 4 or 8, drop or shrink the coarsest levels that only large objects need, and put small anchors, or small assignment ranges for an anchor-free head, on the fine levels. Removing stride-32 outputs also saves head compute.
3. Backbone: keep early, high-resolution stages (less downsampling in the stem) and use a lighter or narrower backbone to pay for it. A pretrained backbone is still worth using; fine-tune it with the first layers frozen (their weights are not updated) if memory is tight. Training stores a layer's input activations because the weight gradient of that layer is computed from them in the backward pass; a frozen layer has no weight gradient, so nothing needs to be stored for it, and if every layer before it is also frozen, no gradient has to flow back through it at all. Only the activations of the trainable layers, and of any layers between them and the loss, need saving.
4. Assignment and loss for tiny boxes. A 1-pixel shift drops the IoU of an 8-pixel box to 0.78 and a 4-pixel shift to 0.33, while the same shifts barely move a 64-pixel box (0.97 and 0.88). A fixed IoU threshold therefore starves small objects of positives; relax the threshold for small boxes, use a centre-distance based assignment (positives are the locations or anchors whose centre lies close to the object's centre, whatever their IoU), or use a loss that still gives a gradient for non-overlapping boxes. A plain IoU loss (1 minus the IoU) is stuck at exactly 1 whenever two boxes do not overlap, however far apart they are, so its gradient is zero and it gives no signal about which way to move; the generalised IoU (GIoU) subtracts from the IoU the share of the smallest box enclosing both boxes that the union does not fill, and the GIoU loss (1 minus GIoU) keeps shrinking as the boxes get closer. Two 8 by 8 boxes placed side by side with a gap of 4, 12 and 40 pixels all have IoU 0.00, but GIoU is -0.20, -0.43 and -0.71 (the enclosing box has area 8 x (16 + gap), the union is 128, so GIoU = -(enclosing - 128) / enclosing), so the GIoU loss still falls as the gap closes and a gradient step moves the prediction toward the object. Add a location-aware term such as an L1 loss on the centre (the sum of absolute differences between the predicted and true centre coordinates).
5. Sample crops around objects. Objects are clustered, so random crops are mostly empty. In the simulation 0.59 of random 512-pixel crops contain no object; drawing half the crops around an object cuts that to 0.30 and raises the mean objects per crop from 5.89 to 16.65. Keep some empty crops (they are the negatives that teach false-positive control).
6. Augmentation that does not destroy small objects: flips, 90-degree rotations (aerial images have no up), colour jitter and mild scale changes; avoid aggressive downscaling or heavy blur. Copy-paste (cut small labelled objects out and paste them onto other crops) adds positives; check the pasted objects look plausible.
7. Detection limits and post-processing: raise the maximum number of detections per tile (hundreds of objects per tile are normal here), tune the score threshold on validation recall, and make NMS (non-maximum suppression: drop lower-scoring boxes that overlap a kept box above a threshold) less aggressive, because neighbouring tiny objects, for example parked cars, genuinely touch.

If you can only do three things, order them by what each one rules out: keep native resolution (items 1 and 2: in the simulation the resize alone left every object under 8 pixels, and a detector cannot recover what the resize removed), sample crops around objects (item 5: the share of empty crops roughly halves, so more steps carry a training signal), then fix the matching rule (item 4: a fixed IoU threshold starves 8-pixel boxes of positives). The printed numbers show how large each problem is, not how much recall each fix buys; that is measured on your own validation tiles. The rest refine a pipeline that already sees the objects.

Fitting into GPU memory

The program estimates activation memory for ONE stored float16 feature map per stage; a real backbone stores several per stage, so read these as ratios, not totals. Memory grows with the crop's pixel count: a 512 crop is 1.8 MiB, 1024 is 7.0 MiB and 2048 is 28.0 MiB for stages at strides 8, 16 and 32. Adding a stride-4 stage with 64 channels raises the 1024 figure to 15.0 MiB (2.14 times), so the finest level is the expensive one. The levers, cheapest first: smaller crops (512 or 768), mixed precision (float16 or bfloat16 arithmetic with float32 master weights), gradient checkpointing (recompute activations in the backward pass instead of storing them, trading compute for memory), smaller batch with gradient accumulation (sum gradients over several steps before updating), and finally a lighter backbone.

import numpy as np
rng = np.random.default_rng(0)

# 1. object size seen by the network when a 4096 px aerial image is resized to the input size
objs = np.exp(rng.uniform(np.log(6), np.log(20), 1000))        # object side lengths in native pixels
print("native median object side: %.1f px" % np.median(objs))
for inp in [1024, 2048, 4096]:
    f = inp / 4096
    print("image resized to %4d: median side %5.1f px | share of objects under 8 px: %.2f" % (inp, np.median(objs) * f, (objs * f < 8).mean()))

# 2. cells covered per side by a 10 px object at each stride
print("10 px object, cells per side at stride 4/8/16/32:", [round(10 / s, 2) for s in [4, 8, 16, 32]])

# 3. IoU of an 8x8 box against a copy shifted by a few pixels (why IoU thresholds punish tiny boxes)
def iou_shift(side, d):
    inter = max(side - d, 0) * side
    return inter / (2 * side * side - inter)
print("IoU of shifted boxes, shift 1/2/4 px: 8 px box", [round(iou_shift(8, d), 2) for d in [1, 2, 4]],
      "| 64 px box", [round(iou_shift(64, d), 2) for d in [1, 2, 4]])

# 4. training crops: objects are clustered, so random crops of a huge image are mostly empty
centres = rng.uniform(500, 3600, (6, 2))
pts = np.vstack([c + rng.normal(0, 150, (50, 2)) for c in centres])        # 300 object centres
pts = np.clip(pts, 0, 4095)
def crop_counts(n, T, mix):
    out = []
    for _ in range(n):
        if rng.random() < mix:                                              # object-centred crop
            c = pts[rng.integers(len(pts))] + rng.uniform(-T / 3, T / 3, 2)
            x0, y0 = np.clip(c - T / 2, 0, 4096 - T)
        else:
            x0, y0 = rng.uniform(0, 4096 - T, 2)
        out.append(((pts[:, 0] >= x0) & (pts[:, 0] < x0 + T) & (pts[:, 1] >= y0) & (pts[:, 1] < y0 + T)).sum())
    return np.array(out)
for mix in [0.0, 0.5]:
    c = crop_counts(2000, 512, mix)
    print("512 px crops, share drawn around an object %.1f: empty crops %.2f, mean objects per crop %.2f" % (mix, (c == 0).mean(), c.mean()))

# 5. activation memory of ONE stored feature map per stage, float16 (2 bytes), batch 1 (arithmetic, not a profile)
def mb(T, s, ch): return (T // s) ** 2 * ch * 2 / 2 ** 20
stages = [(8, 128), (16, 256), (32, 512)]
base = sum(mb(1024, s, c) for s, c in stages)
with4 = base + mb(1024, 4, 64)
print("1024 crop, stages at stride 8/16/32: %.1f MiB; adding a stride-4 stage with 64 channels: %.1f MiB (x%.2f)" % (base, with4, with4 / base))
print("same stages at 512 crop: %.1f MiB, at 2048 crop: %.1f MiB" % (sum(mb(512, s, c) for s, c in stages), sum(mb(2048, s, c) for s, c in stages)))

The complete output:

native median object side: 11.3 px
image resized to 1024: median side   2.8 px | share of objects under 8 px: 1.00
image resized to 2048: median side   5.7 px | share of objects under 8 px: 0.80
image resized to 4096: median side  11.3 px | share of objects under 8 px: 0.22
10 px object, cells per side at stride 4/8/16/32: [2.5, 1.25, 0.62, 0.31]
IoU of shifted boxes, shift 1/2/4 px: 8 px box [0.78, 0.6, 0.33] | 64 px box [0.97, 0.94, 0.88]
512 px crops, share drawn around an object 0.0: empty crops 0.59, mean objects per crop 5.89
512 px crops, share drawn around an object 0.5: empty crops 0.30, mean objects per crop 16.65
1024 crop, stages at stride 8/16/32: 7.0 MiB; adding a stride-4 stage with 64 channels: 15.0 MiB (x2.14)
same stages at 512 crop: 1.8 MiB, at 2048 crop: 28.0 MiB

How to know it worked

Report recall (the share of true objects found) and AP (average precision: the area under a class's precision-recall curve, which summarises how well detections are ranked) by object-size bucket (for example under 8, 8 to 16, 16 to 32 pixels), not only overall mAP (mean AP over classes, often also averaged over several IoU thresholds), because large objects dominate an average. Compare recall at a fixed number of false positives per tile (set the score threshold so the model makes, say, one wrong detection per tile on average, then read the recall; this makes models comparable at the same false-alarm level), and look at tiles where the model misses objects to see whether the cause is the stride, the assignment or the label quality. At inference, tile with overlap so that objects cut by a tile edge are seen whole in a neighbour.

Common mistakes

• Downscaling the image "to fit" and then wondering why recall collapsed; the output above shows every object under 8 pixels at 1024.
• Adding a high-resolution level but leaving the large anchors or the large-object assignment on it.
• Measuring only overall mAP, which hides the small-object recall that the task is about.
• Treating the memory estimate above as a profile. Measure peak memory on the real model.

The idea in plain words

Data augmentation means training on randomly altered copies of each training image (flipped, cropped, recoloured) while keeping the label. The point is to tell the model which changes do not matter. Such a change is called an invariance: a change to the input that should leave the answer the same (a cat moved 10 pixels is still a cat). An epoch is one full pass over the training images; because augmentation redraws its random changes each time an image is loaded, the model sees a different variant of each image in each epoch. If a cat is still a cat when it moves 10 pixels left, an augmentation that moves it teaches the network that position is not evidence for "cat", so the network spends its capacity on features that survive the change. That is why augmentation improves generalization (accuracy on images the model has not seen): the training set effectively contains many plausible variants of every photo, and the model is less able to memorise the exact pixels of the few it has.

The rule for choosing augmentations is one sentence: apply a transform only if it changes the pixels without changing the label, and only in ranges that deployment images could really show. A transform that breaks that rule is the answer to "when does it hurt".

Which augmentations, and what each one teaches

Two rules apply to every row. First, the random augmentation is applied only to the training set; validation and test images are not randomly altered, because a validation score that changes with a random draw cannot be compared between runs. They are, however, a fixed set that looks like deployment: if deployment images are off-centre or dim, the validation set must contain off-centre or dim images (collected, or created once with a fixed seed), otherwise the validation score no longer measures deployment. Second, the random draw happens fresh each time an image is loaded, so one epoch sees different variants than the next.

A measured example, including one that hurts

The program below uses the 8 by 8 handwritten-digit images that ship with scikit-learn (1797 images, 10 classes). A logistic regression on raw pixels is a stand-in for a real convolutional network (a CNN, a network that slides small learned filters over the image), chosen because it trains in seconds; it keeps the experiment honest about what each transform does to the data, but the exact accuracies belong to this toy and will not transfer to a real model. Training uses only 300 images per split (small data is where augmentation matters), repeated over 5 random splits. Each model is scored on a clean test set and on a shifted test set in which every image is moved by up to one pixel in each direction, imitating off-centre objects at deployment.

How the program works, before you read the output. shift(img, dy, dx) moves an image down by dy rows and right by dx columns and fills the vacated edge with zeros; the ys/yd and xs/xd slice pairs pick the source rows and columns and the matching destination rows and columns (for dy = 1 the source rows 0 to 6 are copied to destination rows 1 to 7). shifted_copy applies a random shift of -1, 0 or +1 pixels in each direction to every image. variants builds four training sets: none is the 300 images as they are; shift stacks the originals with two shifted copies (900 images); hflip and vflip stack the originals with their mirror images (600 images). np.concatenate stacks the arrays and np.tile(ytr, reps) repeats the 300 labels once per stacked copy so each image keeps its own label. Each model is then fitted on the stacked set and scored on the clean and the shifted test sets.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression

X, y = load_digits(return_X_y=True)
X = (X / 16.0).reshape(-1, 8, 8)          # 1797 images, 8x8 grayscale, 10 classes

def shift(img, dy, dx):
    out = np.zeros_like(img)
    h, w = img.shape
    ys, yd = (slice(0, h - dy), slice(dy, h)) if dy >= 0 else (slice(-dy, h), slice(0, h + dy))
    xs, xd = (slice(0, w - dx), slice(dx, w)) if dx >= 0 else (slice(-dx, w), slice(0, w + dx))
    out[yd, xd] = img[ys, xs]
    return out

def shifted_copy(imgs, rng):
    return np.stack([shift(im, *rng.integers(-1, 2, size=2)) for im in imgs])

def fit_eval(Xtr, ytr, tests):
    clf = LogisticRegression(max_iter=2000).fit(Xtr.reshape(len(Xtr), -1), ytr)
    return [clf.score(t.reshape(len(t), -1), yt) for t, yt in tests]

rows = {"none": [], "shift": [], "hflip": [], "vflip": []}
for seed in range(5):
    rng = np.random.default_rng(seed)
    idx = rng.permutation(len(X))
    tr, te = idx[:300], idx[300:]          # small training set on purpose
    Xtr, ytr, Xte, yte = X[tr], y[tr], X[te], y[te]
    Xte_shift = shifted_copy(Xte, rng)     # deployment-style jitter the clean set lacks
    tests = [(Xte, yte), (Xte_shift, yte)]
    variants = {
        "none": Xtr,
        "shift": np.concatenate([Xtr, shifted_copy(Xtr, rng), shifted_copy(Xtr, rng)]),
        "hflip": np.concatenate([Xtr, Xtr[:, :, ::-1]]),
        "vflip": np.concatenate([Xtr, Xtr[:, ::-1, :]]),
    }
    for name, Xa in variants.items():
        reps = len(Xa) // len(Xtr)
        rows[name].append(fit_eval(Xa, np.tile(ytr, reps), tests))

print("mean test accuracy over 5 random splits (300 training images each)")
print(f"{'train augmentation':<20}{'clean test':>12}{'shifted test':>14}")
for name, r in rows.items():
    m = np.mean(r, axis=0)
    print(f"{name:<20}{m[0]:>12.3f}{m[1]:>14.3f}")

base = np.array(rows["none"])
for name in ("shift", "hflip", "vflip"):
    r = np.array(rows[name])
    print(f"{name}: lower clean accuracy than none in {int((r[:, 0] < base[:, 0]).sum())} of 5 splits, "
          f"higher shifted accuracy in {int((r[:, 1] > base[:, 1]).sum())} of 5")

Output:

mean test accuracy over 5 random splits (300 training images each)
train augmentation    clean test  shifted test
none                       0.939         0.366
shift                      0.891         0.544
hflip                      0.896         0.328
vflip                      0.881         0.295
shift: lower clean accuracy than none in 5 of 5 splits, higher shifted accuracy in 5 of 5
hflip: lower clean accuracy than none in 5 of 5 splits, higher shifted accuracy in 0 of 5
vflip: lower clean accuracy than none in 5 of 5 splits, higher shifted accuracy in 0 of 5

Reading it:

• Shift augmentation raised accuracy on the shifted test set from 0.366 to 0.544 in all 5 splits. That is the generalization benefit: the model saw the nuisance it will meet later. It cost some clean accuracy here (0.939 to 0.891), because a linear model on raw pixels cannot become shift-invariant, so the shifted copies blur its pixel templates. Do not read that as the price of augmentation in general: it is the price of asking a model with one fixed weight per pixel per class to cover shifted variants. A CNN gets part of shift invariance (more exactly, translation equivariance: when the input moves, its feature maps move with it) from its architecture, so this trade-off is milder there; measure it on your own model rather than assume it. The decision rule that holds up is the one in the next paragraphs: judge an augmentation on a validation set that contains the nuisance, and the shifted test set is that set here.
• Horizontal flip lowered clean accuracy (0.939 to 0.896) and did not help the shifted set (0.328 against 0.366 with no augmentation). Digits are not mirror-symmetric, so a flipped copy is not a faithful example of its class and the label no longer describes the content. Flip is a label-changing transform for this data, so the model is trained on contradictions.
• Vertical flip is worse still (0.881 clean), and it also shows the classic pair: an upside-down 6 is a 9.

A worked blend: how mixup and CutMix change the label

Mixup builds a training example from two images with weight $\lambda$ (here 0.7 for the first image): pixels are $0.7 \cdot A + 0.3 \cdot B$ and the label is $0.7 \cdot \text{label}_A + 0.3 \cdot \text{label}_B$, a vector that is 70% "cat" and 30% "dog" instead of a single class. CutMix pastes a rectangle of B into A and sets the label weights to the area fractions (a quarter of the image pasted, so 0.75 and 0.25). The loss must then be the soft-label cross-entropy, $-\sum_k y_k \log p_k$, which compares the model's probabilities with the blended label. The tiny program below does both blends on 4 by 4 arrays and shows how the loss differs for the same prediction.

import numpy as np

# Two tiny 4x4 "images" (a cat image of 1s and a dog image of 0s) and their one-hot labels over [cat, dog, bird].
cat, dog = np.ones((4, 4)), np.zeros((4, 4))
y_cat, y_dog = np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 0.0])

# Mixup: blend whole images and labels with the same weight lam.
lam = 0.7
mix_img = lam * cat + (1 - lam) * dog
mix_lab = lam * y_cat + (1 - lam) * y_dog
print("mixup pixel value everywhere:", mix_img[0, 0], "| label:", mix_lab.round(2).tolist())

# CutMix: paste a 2x2 patch of the dog image into the cat image; the label weight is the area fraction.
cut = cat.copy(); cut[:2, :2] = dog[:2, :2]
frac_dog = (cut == 0).mean()
cut_lab = (1 - frac_dog) * y_cat + frac_dog * y_dog
print("cutmix dog area fraction:", frac_dog, "| label:", cut_lab.round(2).tolist())

# Soft-label loss: cross-entropy against the blended label instead of a single class.
p = np.array([0.6, 0.3, 0.1])                      # the model's predicted probabilities for [cat, dog, bird]
print("loss against hard label cat:", round(float(-np.log(p[0])), 3))
print("loss against mixup label   :", round(float(-(mix_lab * np.log(p)).sum()), 3))

Output:

mixup pixel value everywhere: 0.7 | label: [0.7, 0.3, 0.0]
cutmix dog area fraction: 0.25 | label: [0.75, 0.25, 0.0]
loss against hard label cat: 0.511
loss against mixup label   : 0.719

Reading it: a mixed image has every pixel at 0.7 (between the all-1 cat and all-0 dog) and its label has 0.7 on cat and 0.3 on dog. The CutMix image has 4 of 16 pixels from the dog, so its label is 0.75 / 0.25. For a model that predicts cat 0.6, dog 0.3, bird 0.1, the loss against the hard label is 0.511 but against the mixup label is 0.719: the cheapest prediction for the blended label is the mixture itself (0.7 cat, 0.3 dog, nothing on bird), so a model that answers 99% cat is penalised, which is why mixup discourages over-confidence.

When an augmentation hurts

An augmentation hurts when it changes the label, or produces images deployment never contains. Concrete cases a strong answer names:

1. Flips on orientation-bearing content: digits and text (b and d, 6 and 9), road signs (left turn vs right turn), medical scans where the left/right side carries meaning, satellite images where north is fixed.
2. Hue or colour jitter when colour is the label: ripe vs unripe fruit, traffic light state, skin lesion colour.
3. Aggressive crops that remove the object that defines the label (a crop of the background of a "dog on a sofa" image labelled "dog").
4. Rotation by large angles on data with a fixed camera angle: it teaches invariance to views that never occur and uses model capacity for them.
5. Blur or noise heavier than reality: it can erase the fine texture that separates two classes (for example, fine defects on a manufactured part).

How to choose in practice

Start from the deployment conditions (camera, lighting, framing), list the nuisances that really vary, and augment for exactly those. Then validate: add one augmentation at a time, compare against a validation set that contains the variation deployment will contain, and keep it only if it helps there. If deployment really is clean and centred, that validation set is the clean one and a transform that lowers it should go; if deployment is off-centre, the clean score alone would wrongly reject the shift augmentation (it fell from 0.939 to 0.891 on clean images but rose from 0.366 to 0.544 on shifted ones, the images deployment resembles), so the shifted set is the one that decides. Look at a grid of augmented samples with their labels before training; mislabelled-by-augmentation images are visible by eye.

Common mistakes

• Augmenting the validation or test set, which hides overfitting.
• Applying a transform that changes the label (the flip case above) because it is "standard".
• Treating augmentation as a substitute for more data in a different distribution; it can only vary what is already there.
• Forgetting that stronger augmentation usually needs longer training to converge.