  The transfer of tensors from/to memory during neural network training
dominates time and energy. To improve energy efficiency and performance,
research has been exploring ways to use narrower data representations. So far,
these attempts relied on user-directed trial-and-error to achieve convergence.
We present methods that relieve users from this responsibility. Our methods
dynamically adjust the size and format of the floating-point containers used
for activations and weights during training, achieving adaptivity across three
dimensions: i) which datatype to use, ii) on which tensor, and iii) how it
changes over time. The different meanings and distributions of exponent and
mantissas lead us to tailored approaches for each. We present two lossy pairs
of methods to eliminate as many mantissa and exponent bits as possible without
affecting accuracy. Quantum Mantissa and Quantum Exponent are machine learning
compression methods that tap into the gradient descent algorithm to learn the
minimal mantissa and exponent bitlengths on a per-layer granularity. They
automatically learn that many tensors can use just 1 or 2 mantissa bits and 3
or 4 exponent bits. Overall, the two machine learning methods reduce the
footprint by $4.74\times$. Alternatively, BitWave observes changes in the loss
function during training to adjust mantissa and exponent bitlengths
network-wide, yielding a $3.19\times$ reduction in footprint. Finally, we
present an optional method, Gecko, to exploit the naturally emerging, lop-sided
exponent distribution to losslessly compress resulting exponents from Quantum
Exponent or BitWave and, on average, improve compression rates to $5.64\times$
and $4.56\times$.
