  Training convolutional neural network models is memory intensive since
back-propagation requires storing activations of all intermediate layers. This
presents a practical concern when seeking to deploy very deep architectures in
production, especially when models need to be frequently re-trained on updated
datasets. In this paper, we propose a new implementation for back-propagation
that significantly reduces memory usage, by enabling the use of approximations
with negligible computational cost and minimal effect on training performance.
The algorithm reuses common buffers to temporarily store full activations and
compute the forward pass exactly. It also stores approximate per-layer copies
of activations, at significant memory savings, that are used in the backward
pass. Compared to simply approximating activations within standard
back-propagation, our method limits accumulation of errors across layers. This
allows the use of much lower-precision approximations without affecting
training accuracy. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that
our method yields performance close to exact training, while storing
activations compactly with as low as 4-bit precision.
