Data augmentation

Two ways into the same idea

Before You Start

The Core Idea

Augmentation makes new examples from old ones. Here you learn the question that comes before every transform, then safe and destructive transforms, then measuring the effect, then where augmentation is applied.

1The one question: does the answer stay?

Before any transform, ask: if I apply it to the image, does the right answer stay the same? A horizontally flipped cat is still a cat, but a flipped 2 is not a 2.

import numpy as np

two = np.array([
    [0, 1, 1, 0],
    [1, 0, 0, 1],
    [0, 0, 1, 0],
    [0, 1, 0, 0],
    [1, 1, 1, 1],
])
for name, img in [("original", two), ("flipped", two[:, ::-1]), ("brighter", np.clip(two * 1.0 + 0.3, 0, 1))]:
    print(name)
    for row in img:
        print("  " + "".join("#" if v >= 1 else ("+" if v > 0 else ".") for v in row))

The flipped 2 became a shape like no digit, its diagonal now running the other way. Raising brightness kept the shape exactly. That is the difference between helpful and destructive augmentation, and the decision depends on the task, not the library.

2Safe and destructive transforms

The decision follows the task, not the library. When direction or shape is the answer, as with text, digits and direction signs, flipping becomes an error.

Look at the picture: one question splits tasks into two branches.

Random crops and resizing, slight brightness and contrast changes, light noise and small rotations are safe in most tasks. Cropping is usually the most useful.

Beware colour changes when colour itself is the answer, like fruit ripeness. And in medical images, left versus right may be the whole difference.

3Measure every transform

Augmentation puts your knowledge into the model. A right rule saves it thousands of examples, and a wrong rule teaches it a mistake just as strongly.

So we treat every transform like any feature: train with it and without it, and compare on fixed validation.

Only 200 digit images for training, and test data shifted one pixel left or right at random, as if the camera was not perfectly centred. Three trainings: no augmentation, random shifts, and horizontal flips.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(digits.images, digits.target, train_size=200, random_state=0, stratify=digits.target)
rng = np.random.default_rng(0)
X_te_shifted = np.array([np.roll(img, rng.integers(-1, 2), axis=1) for img in X_te])


def shift(img):
    return np.roll(img, rng.integers(-1, 2), axis=1)


def flip(img):
    return img[:, ::-1]


def fit_and_score(extra=None):
    imgs, labels = list(X_tr), list(y_tr)
    if extra:
        imgs += [extra(img) for img in X_tr]
        labels += list(y_tr)
    model = LogisticRegression(max_iter=3000).fit(np.array(imgs).reshape(len(imgs), -1), labels)
    return model.score(X_te_shifted.reshape(len(X_te), -1), y_te)


print(f"no augmentation:    {fit_and_score():.3f}")
print(f"random 1px shifts:  {fit_and_score(shift):.3f}")
print(f"horizontal flips:   {fit_and_score(flip):.3f}")

No augmentation: 56.2%. The model memorised digits in their exact position, so one pixel of shift ruined half its predictions. Random shifts lifted it to 80.1%, because they taught exactly what it needed: position does not matter. Horizontal flips dropped it to 54.8%, worse than nothing, because they taught it a digit and its mirror image are the same class. Same data, same model, and the difference is one augmentation decision.

A transform that does not improve the number goes, however sensible it looks.

4On training data only

Augmentation is random, changes every epoch, and touches training data only. Validation and test always go through the same fixed processing.

Look at the picture: only the training part is random. Apply it to validation and you are measuring a moving target.

from torchvision import transforms

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.RandomRotation(8),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
eval_tf = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

Always two separate pipelines: random for training, fixed for evaluation. Note the absence of RandomHorizontalFlip: this pipeline is for images containing Arabic text or digits. Note too the rotation is only 8 degrees, since 90 would turn a 6 into a 9.

How It Works, Step by Step: how to choose augmentation that helps rather than hurts

Worked example

The only question before any transform: if I apply it to an image, does the right answer stay the same? A horizontally flipped cat is still a cat. A flipped 2 is not a 2.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding