  Training wide and deep neural networks (DNNs) require large amounts of
storage resources such as memory because the intermediate activation data must
be saved in the memory during forward propagation and then restored for
backward propagation. However, state-of-the-art accelerators such as GPUs are
only equipped with very limited memory capacities due to hardware design
constraints, which significantly limits the maximum batch size and hence
performance speedup when training large-scale DNNs. Traditional memory saving
techniques either suffer from performance overhead or are constrained by
limited interconnect bandwidth or specific interconnect technology. In this
paper, we propose a novel memory-efficient CNN training framework (called
COMET) that leverages error-bounded lossy compression to significantly reduce
the memory requirement for training, to allow training larger models or to
accelerate training. Different from the state-of-the-art solutions that adopt
image-based lossy compressors (such as JPEG) to compress the activation data,
our framework purposely adopts error-bounded lossy compression with a strict
error-controlling mechanism. Specifically, we perform a theoretical analysis on
the compression error propagation from the altered activation data to the
gradients, and empirically investigate the impact of altered gradients over the
training process. Based on these analyses, we optimize the error-bounded lossy
compression and propose an adaptive error-bound control scheme for activation
data compression. We evaluate our design against state-of-the-art solutions
with five widely-adopted CNNs and ImageNet dataset. Experiments demonstrate
that our proposed framework can significantly reduce the training memory
consumption by up to 13.5X over the baseline training and 1.8X over another
state-of-the-art compression-based framework, respectively, with little or no
accuracy loss.
