In this post we’ll go over:

Image Augmentation: What is it? Why is it important?

Keras: How to use it for basic Image Augmentation.

Histogram Equalization: What is it? How is it useful?

Implementing Histogram Equalization Techniques: one way to modify the keras.preprocessing image.py file.

Image Augmentation: What is it? Why is it important?

Deep Neural Networks, particularly Convolutional Neural Networks (CNNs), are particularly proficient at image classification tasks. State-of-the-art CNNs have even been shown to exceed human performance in image recognition.

However, as we learn from Mr. Jian-Yang’s “Hot Dog, Not Hot Dog” food recognition app in the popular TV show, Silicon Valley, (the app is now available on the app store) gathering images as training data can be both costly and time consuming.

If you aren’t familiar with the TV show Silicon Valley, be warned that the language in the following clip is NSFW:

In order to combat the high expense of collecting thousands of training images, image augmentation has been developed in order to generate training data from an existing dataset. Image Augmentation is the process of taking images that are already in a training dataset and manipulating them to create many altered versions of the same image. This both provides more images to train on, but can also help expose our classifier to a wider variety of lighting and coloring situations so as to make our classifier more robust. Here are some examples of different augmentations from the imgaug library.

Using Keras for Basic Image Augmentation

There are many ways to pre-process images. In this post we will go over some of the most common out-of-the-box methods that the keras deep learning library provides for augmenting images, then we will show how to alter the keras.preprocessing image.py file in order to enable histogram equalization methods. We will use the cifar10 dataset that comes with keras. However, we will only be using the images of cats and dogs from the dataset in order to keep the task small enough to be performed on a CPU — in case you want to follow along. You can view an IPython notebook of the source code from this post.

Loading and Formatting the Data

The first thing that we will do is load the cifar10 dataset and format the images to prepare them for the CNN. We’ll also take a peek at a few of the images just to make sure the data has loaded properly.

The cifar10 images are only 32 x 32 pixels, so they look grainy when magnified here, but the CNN doesn’t know it’s grainy, all it sees is DATA.

Create an image generator from ImageDataGenerator()

Augmenting our image data with keras is dead simple. A shoutout to Jason Brownlee who provides a great tutorial on this. First we need to create an image generator by calling the ImageDataGenerator() function and pass it a list of parameters describing the alterations that we want it to perform on the images. We will then call the fit() function on our image generator which will apply the changes to the images batch by batch. By default, the modifications will be applied randomly, so not every image will be changed every time. You can also use keras.preprocessing to export augmented image files to a folder in order to build up a giant dataset of altered images should you desire to do so.

We’ll look at some of the more visually interesting augmentations here. A description of the all of the possible ImageDataGenerator() parameters as well as a list of the other methods available in keras.preprocessing can be seen in the keras documentation.

Randomly Rotate Images

Flip Images Vertically

Flipping images horizontally is also one of the classic ways of generating more data for a classifier. It is just as easy to do and probably makes more sense with this dataset, however, I’ve left out the code and images because there’s no way of knowing whether a dog or cat image has been flipped horizontally without seeing the original.

Shift Images Vertically or Horizontally by 20%

Histogram Equalization Techniques

Histogram Equalization is the process taking a low contrast image and increasing the contrast between the image’s relative highs and lows in order to bring out subtle differences in shade and create a higher contrast image. The results can be striking, especially for grayscale images. Here are some examples:

In this post we’ll be looking at three image augmentation techniques for improving contrast in images. These approaches are sometimes also referred to as “Histogram Stretching” because they take the distribution of pixel intensities and stretch the distribution to fit a wider range of values thereby increasing the level of contrast between the lightest and darkest portions of an image.

Histogram Equalization

Histogram Equalization increases contrast in images by detecting the distribution of pixel densities in an image and plotting these pixel densities on a histogram. The distribution of this histogram is then analyzed and if there are ranges of pixel brightnesses that aren’t currently being utilized, the histogram is then “stretched” to cover those ranges, and then is “back projected” onto the image to increase the overall contrast of the image.

Contrast Stretching

Contrast Stretching takes the approach of analyzing the distribution of pixel densities in an image and then “rescales the image to include all intensities that fall within the 2nd and 98th percentiles.”

Adaptive Equalization

Adaptive Equalization differs from regular histogram equalization in that several different histograms are computed, each corresponding to a different section of the image; however, it has a tendency to over-amplify noise in otherwise uninteresting sections.

The following code comes from the sci-kit image library’s docs and has been altered to perform the three above augmentations on the first image of our cifar10 dataset. First we will import the necessary modules from the sci-kit image (skimage) library and then modify the code from the sci-kit image documentation to view the augmentations on the first image of our dataset.

Here are the modified images of a low contrast cat from the cifar10 dataset. As you can see, the results are not as striking as they might be with a low contrast grayscale image, but still help improve the quality of the images.

Modifying keras.preprocessing to enable Histogram Equalization techniques.

Now that we have successfully modified one image from the cifar10 dataset, we will demonstrate how to alter the keras.preprocessing image.py file in order to execute these different histogram modification techniques just as we did the out-of-the-box keras augmentations using ImageDataGenerator() . Here are the general steps that we will follow in order to implement this functionality:

Overview

Find the keras.preprocessing image.py file on your own machine.

Copy the image.py file into your file or notebook.

Add one attribute for each equalization technique to the DataImageGenerator() init function.

Add IF statement clauses to the random_transform method so that augmentations get implemented when we call datagen.fit() .

One of the simplest ways to make alterations to keras.preprocessing’s image.py file is simply to copy and paste its contents into our code. This will then remove the need to import it. You can view the contents of the image.py file on github here. However, in order to be sure that you’re grabbing the same version of the file that you were importing previously, it’s better to grab the image.py file that is already on your machine. Running print(keras.__file__) will print out the path to the keras library that is on your machine. The path (for mac users) may look something like:

/usr/local/lib/python3.5/dist-packages/keras/__init__.pyc

This gives us the path to keras on our local machine. Go ahead and navigate there and then into the preprocessing folder. Inside preprocessing you will see the image.py file. You can then copy its contents into your code. The file is long, but for beginners this is probably one of the easiest ways to go about making alterations to it.

Editing image.py

At the top of image.py you can comment out the line: from ..import backend as K if you have already included it above.

At this point, also double-check to make sure that you’re importing the necessary scikit-image modules so that the copied image.py can see them.

from skimage import data, img_as_float

from skimage import exposure

We now need to add six lines to the ImageDataGenerator class’s __init__ method so that it has the three properties that represent the types of augmentation that we’re going to be adding. The code below is copied from my current image.py. The lines with ##### to the side are lines that I have added.

The random_transform() function (below) responds to the arguments we have been passed into the ImageDataGenerator() function. If we have set the contrast_stretching , adaptive_equalization , or histogram_equalization parameters to True , when we call ImageDataGenerator() , (just like we would for the other image augmentations) random_transform() will then apply the desired image augmentation.

Now we have all of the necessary code in place and can call ImageDataGenerator() to perform our histogram modification techniques. Here’s what a few images look like if we set all three of the values to True .

I don’t recommend setting more than one of them to True for any given dataset. Make sure you experiment with your particular dataset in order to see what helps improve your classifier’s accuracy. For colored images, I’ve found that contrast stretching often obtains better results than histogram modification or adaptive equalization.

Train and Validate your Keras CNN