  We present FPRaker, a processing element for composing training accelerators.
FPRaker processes several floating-point multiply-accumulation operations
concurrently and accumulates their result into a higher precision accumulator.
FPRaker boosts performance and energy efficiency during training by taking
advantage of the values that naturally appear during training. Specifically, it
processes the significand of the operands of each multiply-accumulate as a
series of signed powers of two. The conversion to this form is done on-the-fly.
This exposes ineffectual work that can be skipped: values when encoded have few
terms and some of them can be discarded as they would fall outside the range of
the accumulator given the limited precision of floating-point. We demonstrate
that FPRaker can be used to compose an accelerator for training and that it can
improve performance and energy efficiency compared to using conventional
floating-point units under ISO-compute area constraints. We also demonstrate
that FPRaker delivers additional benefits when training incorporates pruning
and quantization. Finally, we show that FPRaker naturally amplifies performance
with training methods that use a different precision per layer.
