  Training Convolutional Neural Networks (CNN) is a resource intensive task
that requires specialized hardware for efficient computation. One of the most
limiting bottleneck of CNN training is the memory cost associated with storing
the activation values of hidden layers needed for the computation of the
weights gradient during the backward pass of the backpropagation algorithm.
Recently, reversible architectures have been proposed to reduce the memory cost
of training large CNN by reconstructing the input activation values of hidden
layers from their output during the backward pass, circumventing the need to
accumulate these activations in memory during the forward pass. In this paper,
we push this idea to the extreme and analyze reversible network designs
yielding minimal training memory footprint. We investigate the propagation of
numerical errors in long chains of invertible operations and analyze their
effect on training. We introduce the notion of pixel-wise memory cost to
characterize the memory footprint of model training, and propose a new model
architecture able to efficiently train arbitrarily deep neural networks with a
minimum memory cost of 352 bytes per input pixel. This new kind of architecture
enables training large neural networks on very limited memory, opening the door
for neural network training on embedded devices or non-specialized hardware.
For instance, we demonstrate training of our model to 93.3% accuracy on the
CIFAR10 dataset within 67 minutes on a low-end Nvidia GTX750 GPU with only 1GB
of memory.
