  Bayesian Neural Networks (BNNs) that possess a property of uncertainty
estimation have been increasingly adopted in a wide range of safety-critical AI
applications which demand reliable and robust decision making, e.g.,
self-driving, rescue robots, medical image diagnosis. The training procedure of
a probabilistic BNN model involves training an ensemble of sampled DNN models,
which induces orders of magnitude larger volume of data movement than training
a single DNN model. In this paper, we reveal that the root cause for BNN
training inefficiency originates from the massive off-chip data transfer by
Gaussian Random Variables (GRVs). To tackle this challenge, we propose a novel
design that eliminates all the off-chip data transfer by GRVs through the
reversed shifting of Linear Feedback Shift Registers (LFSRs) without incurring
any training accuracy loss. To efficiently support our LFSR reversion strategy
at the hardware level, we explore the design space of the current DNN
accelerators and identify the optimal computation mapping scheme to best
accommodate our strategy. By leveraging this finding, we design and prototype
the first highly efficient BNN training accelerator, named Shift-BNN, that is
low-cost and scalable. Extensive evaluation on five representative BNN models
demonstrates that Shift-BNN achieves an average of 4.9x (up to 10.8x) boost in
energy efficiency and 1.6x (up to 2.8x) speedup over the baseline DNN training
accelerator.
