How to Improve Model Performance With Image Augmentation

Data preparation is required when working with neural network and deep learning models. Increasingly data augmentation is also required on more complex object recognition tasks. In this lesson you will discover how to use data preparation and data augmentation with your image datasets when developing and evaluating deep learning models in Python with Keras. After completing this lesson, you will know:

• About the image augmentation API provide by Keras and how to use it with your models.
• How to perform feature standardization.
• How to perform ZCA whitening of your images.
• How to augment data with random rotations, shifts and flips of images.
• How to save augmented image data to disk.

1.1 Keras Image Augmentation API

Like the rest of Keras, the image augmentation API is simple and powerful. Keras provides the ImageDataGenerator class that defines the configuration for image data preparation and augmentation. This includes capabilities such as:

• Feature-wise standardization.
• ZCA whitening
• Random rotation, shifts, shear and flips.
• Dimension reordering.
• Save augmented images to disk.

An augmented image generator can be created as follows:

# Create a ImageDataGenerator

datagen = ImageDataGenerator()

Rather than performing the operations on your entire image dataset in memory, the API is designed to be iterated by the deep learning model fitting process, creating augmented image data for you just-in-time. This reduces your memory overhead, but adds some additional time cost during model training. After you have created and configured your ImageDataGenerator, you must fit it on your data. This will calculate any statistics required to actually perform the transforms to your image data. You can do this by calling the fit() function on the data generator and pass it your training dataset.

# Fit the ImageDataGenerator

datagen.fit(train)

The data generator itself is in fact an iterator, returning batches of image samples when requested. We can configure the batch size and prepare the data generator and get batches of images by calling the flow() function.

# Configure the Batch Size for the ImageDataGenerator

X_batch, y_batch = datagen.flow(train, train, batch_size=32)

Finally we can make use of the data generator. Instead of calling the fit() function on our model, we must call the fit generator() function and pass in the data generator and the desired length of an epoch as well as the total number of epochs on which to train.

# Fit a Model Using the ImageDataGenerator

fit_generator(datagen, samples_per_epoch=len(train), epochs=100)

1.2 Point of Comparison for Image Augmentation

Now that you know how the image augmentation API in Keras works, let’s look at some examples. We will use the MNIST handwritten digit recognition task in these examples (learn more in Section 19.1). To begin, let’s take a look at the first 9 images in the training dataset.

# Load and Plot the MNIST dataset
# Plot of images as baseline for comparison
from keras.datasets import mnist
from matplotlib import pyplot
# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()
# create a grid of 3 x 3 images
for i in range(0, 9):
    pyplot.subplot(330 + 1 + i)
    pyplot.imshow(X_train[i], cmap=pyplot.get_cmap('gray'))
    
# show the plot 
pyplot.show()

 

 Running this example provides the following image that we can use as a point of comparison with the image preparation and augmentation tasks in the examples below.

 1.3 Feature Standardization

It is also possible to standardize pixel values across the entire dataset. This is called feature standardization and mirrors the type of standardization often performed for each column in a tabular dataset. This is di↵erent to sample standardization described in the previous section as pixel values are standardized across all samples (all images in the dataset). In this case each image is considered a feature. You can perform feature standardization by setting the featurewise_center and featurewise_std_normalization arguments on the ImageDataGenerator class.

# Example of Feature Standardization
# Standardize images across the dataset, mean=0, stdev=1
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot
# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()
# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)

# convert from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation
datagen = ImageDataGenerator(featurewise_center=True, featurewise_std_normalization=True)

# fit parameter from data
datagen.fit(X_train)
# configure batch size and retrive one batch of images
for X_batch, y_batch in datagen.flow(X_train, y_train, batch_size=9):
    # create a grid of 3 X 3 images
    for i in range(0,9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28,28), cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break
    

 

Running this example you can see that the e↵ect on the actual images, seemingly darkening and lightening di↵erent digits.

1.4 ZCA Whitening

        A whitening transform of an image is a linear algebra operation that reduces the redundancy in the matrix of pixel images. Less redundancy in the image is intended to better highlight the structures and features in the image to the learning algorithm. Typically, image whitening is performed using the Principal Component Analysis (PCA) technique. More recently, an alternative called ZCA (learn more in Appendix A of this tech report2) shows better results and results in transformed images that keeps all of the original dimensions and unlike PCA, resulting transformed images still look like their originals. You can perform a ZCA whitening transform by setting the zca whitening argument to True.

# Example of ZCA Whitening
# ZCA whitening
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot

# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()
# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)

# convert from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation 
datagen = ImageDataGenerator(zca_whitening=True)
# fit parameters from data
datagen.fit(X_train)
# configure batch size and retrieve one batch of images
for X_batch, y_batch in datagen.flow(X_train, y_train, batch_size=9):
    # create a grid of 3x3 images
    for i in range(0, 9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28, 28), cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break

 Running the example, you can see the same general structure in the images and how the outline of each digit has been highlighted.

 1.5 Random Rotations

Sometimes images in your sample data may have varying and different rotations in the scene. You can train your model to better handle rotations of images by artificially and randomly rotating images from your dataset during training. The example below creates random rotations of the MNIST digits up to 90 degrees by setting the rotation range argument.

# Example of Random Image Rotations
# Random Rotations
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot
# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()
# reshape to be [samples][pixeles][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)

# convert  from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation
datagen = ImageDataGenerator(rotation_range=90)

# fit parameters from data

datagen.fit(X_train)
# configure batch size and retrive one batch of images
for X_batch,y_batch in datagen.flow(X_train,y_train,batch_size=9):
    # create a grid of 3 X 3 images
    for i in range(0, 9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28, 28), cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break

Running the example, you can see that images have been rotated left and right up to a limit of 90 degrees. This is not helpful on this problem because the MNIST digits have a normalized orientation, but this transform might be of help when learning from photographs where the objects may have different orientations.

 1.6 Random Shifts

Objects in your images may not be centered in the frame. They may be off-center in a variety of different ways. You can train your deep learning network to expect and currently handle off-center objects by artificially creating shifted versions of your training data. Keras supports separate horizontal and vertical random shifting of training data by the width_shift_range and height_shift_range arguments.

# Example of Random Image Shifts
# Random Shifts
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)
# convert from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation
shift = 0.2
datagen = ImageDataGenerator(width_shift_range=shift, height_shift_range=shift)
# fit parameters from data
datagen.fit(X_train)
# configure batch size and retrieve one batch of images
for X_batch, y_batch in datagen.flow(X_train,y_train, batch_size=9):
    # create a grid of 3X3 images
    for i in range(0, 9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28, 28),cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break

Running this example creates shifted versions of the digits. Again, this is not required for MNIST as the handwritten digits are already centered, but you can see how this might be useful on more complex problem domains

1.7 Random Flips

Another augmentation to your image data that can improve performance on large and complex problems is to create random flips of images in your training data. Keras supports random flipping along both the vertical and horizontal axes using the vertical flip and horizontal flip arguments.

# Example of Random Image Flips
# Random Flips
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot

# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()

# reshape to be [sample][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)

# convert from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation
datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True)

# fit parameters from data
datagen.fit(X_train)
# configure batch size and retrieve one batch of images

for X_batch,y_batch in datagen.flow(X_train, y_train,batch_size=9):
    # create a grid of 3X3 images
    for i in range(0, 9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28, 28), cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break

Running this example you can see flipped digits. Flipping digits in MNIST is not useful as they will always have the correct left and right orientation, but this may be useful for problems with photographs of objects in a scene that can have a varied orientation.

1.8 Saving Augmented Images to File

The data preparation and augmentation is performed just-in-time by Keras. This is efficient in terms of memory,but you may require the exact images used during training. For example, perhaps you would like to use them with a different software package later or only generate them once and use them on multiple di↵erent deep learning models or configurations.

        Keras allows you to save the images generated during training. The directory, filename prefix and image file type can be specified to the flow() function before training. Then, during training, the generated images will be written to file. The example below demonstrates this and writes 9 images to a images subdirectory with the prefix aug and the file type of PNG.

# Save augmented images to file
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot
import os
from keras import backend as K
K.set_image_data_format('channels_first')
# load data
(X_train, y_train),(X_test, y_test) = mnist.load_data()
# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)

# convert from int to float
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# define data preparation
datagen = ImageDataGenerator()

# fit parameters from data
datagen.fit(X_train)

# configure batch size and retrieve  one batch of images
os.makedirs('images')
for X_batch, y_batch in datagen.flow(X_train, y_train, batch_size=9, save_to_dir='images',save_prefix='aug',save_format='png'):
    #create a grid of 3x3 images
    for i in range(0, 9):
        pyplot.subplot(330 + 1 + i)
        pyplot.imshow(X_batch[i].reshape(28, 28), cmap=pyplot.get_cmap('gray'))
    # show the plot
    pyplot.show()
    break

 Running the example you can see that images are only written when they are generated.

 1.9 Tips For Augmenting Image Data with Keras

 Image data is unique in that you can review the transformed copies of the data and quickly get an idea of how the model may be perceive it by your model. Below are some times for getting the most from image data preparation and augmentation for deep learning.

• Review Dataset. Take some time to review your dataset in great detail. Look at the images. Take note of image preparation and augmentations that might benefit the training process of your model, such as the need to handle di↵erent shifts, rotations or flips of objects in the scene.
• Review Augmentations. Review sample images after the augmentation has been performed. It is one thing to intellectually know what image transforms you are using, it is a very di↵erent thing to look at examples. Review images both with individual augmentations you are using as well as the full set of augmentations you plan to use in aggregate. You may see ways to simplify or further enhance your model training process.
• Evaluate a Suite of Transforms. Try more than one image data preparation and augmentation scheme. Often you can be surprised by results of a data preparation scheme you did not think would be beneficial.

1.10 Summary

In this lesson you discovered image data preparation and augmentation. You discovered a range of techniques that you can use easily in Python with Keras for deep learning models. You learned about:

• The ImageDataGenerator API in Keras for generating transformed images just-in-time.
• Feature-wise pixel standardization.
• The ZCA whitening transform.
• Random rotations, shifts and flips of images.
• How to save transformed images to file for later reuse.

1.10.1 Next

You now know how to develop convolutional neural networks and use the image augmentation API in Keras. In the next chapter you will work through developing larger and deeper models for a more complex object recognition task using Keras