  Modern deep neural network (DNN) models generally require a huge amount of
weight and activation values to achieve good inference outcomes. Those data
inevitably demand a massive off-chip memory capacity/bandwidth, and the
situation gets even worse if they are represented in high-precision
floating-point formats. Effort has been made for representing those data in
different 8-bit floating-point formats, nevertheless, a notable accuracy loss
is still unavoidable. In this paper we introduce an extremely flexible 8-bit
floating-point (FFP8) format whose defining factors - the bit width of
exponent/fraction field, the exponent bias, and even the presence of the sign
bit - are all configurable. We also present a methodology to properly determine
those factors so that the accuracy of model inference can be maximized. The
foundation of this methodology is based on a key observation - both the maximum
magnitude and the value distribution are quite dissimilar between weights and
activations in most DNN models. Experimental results demonstrate that the
proposed FFP8 format achieves an extremely low accuracy loss of $0.1\%\sim
0.3\%$ for several representative image classification models even without the
need of model retraining. Besides, it is easy to turn a classical
floating-point processing unit into an FFP8-compliant one, and the extra
hardware cost is minor.
