  Conventional hardware-friendly quantization methods, such as fixed-point or
integer, tend to perform poorly at very low word sizes as their shrinking
dynamic ranges cannot adequately capture the wide data distributions commonly
seen in sequence transduction models. We present AdaptivFloat, a floating-point
inspired number representation format for deep learning that dynamically
maximizes and optimally clips its available dynamic range, at a layer
granularity, in order to create faithful encoding of neural network parameters.
AdaptivFloat consistently produces higher inference accuracies compared to
block floating-point, uniform, IEEE-like float or posit encodings at very low
precision ($\leq$ 8-bit) across a diverse set of state-of-the-art neural
network topologies. And notably, AdaptivFloat is seen surpassing baseline FP32
performance by up to +0.3 in BLEU score and -0.75 in word error rate at weight
bit widths that are $\leq$ 8-bit. Experimental results on a deep neural network
(DNN) hardware accelerator, exploiting AdaptivFloat logic in its computational
datapath, demonstrate per-operation energy and area that is 0.9$\times$ and
1.14$\times$, respectively, that of equivalent bit width integer-based
accelerator variants.
