  Bit-serial architectures can handle Neural Networks (NNs) with different
weight precisions, achieving higher resource efficiency compared with
bit-parallel architectures. Besides, the weights contain abundant zero bits
owing to the fault tolerance of NNs, indicating that bit sparsity of NNs can be
further exploited for performance improvement. However, the irregular
proportion of zero bits in each weight causes imbalanced workloads in the
Processing Element (PE) array, which degrades performance or induces overhead
for sparse processing. Thus, this paper proposed a bit-sparsity quantization
method to maintain the bit sparsity ratio of each weight to no more than a
certain value for balancing workloads, with little accuracy loss. Then, we
co-designed a sparse bit-serial architecture, called Bit-balance, to improve
overall performance, supporting weight-bit sparsity and adaptive bitwidth
computation. The whole design was implemented with 65nm technology at 1 GHz and
performs at 326-, 30-, 56-, and 218-frame/s for AlexNet, VGG-16, ResNet-50, and
GoogleNet respectively. Compared with sparse bit-serial accelerator, Bitlet,
Bit-balance achieves 1.8x~2.7x energy efficiency (frame/J) and 2.1x~3.7x
resource efficiency (frame/mm2).
