  Network binarization emerges as one of the most promising compression
approaches offering extraordinary computation and memory savings by minimizing
the bit-width. However, recent research has shown that applying existing
binarization algorithms to diverse tasks, architectures, and hardware in
realistic scenarios is still not straightforward. Common challenges of
binarization, such as accuracy degradation and efficiency limitation, suggest
that its attributes are not fully understood. To close this gap, we present
BiBench, a rigorously designed benchmark with in-depth analysis for network
binarization. We first carefully scrutinize the requirements of binarization in
the actual production and define evaluation tracks and metrics for a
comprehensive and fair investigation. Then, we evaluate and analyze a series of
milestone binarization algorithms that function at the operator level and with
extensive influence. Our benchmark reveals that 1) the binarized operator has a
crucial impact on the performance and deployability of binarized networks; 2)
the accuracy of binarization varies significantly across different learning
tasks and neural architectures; 3) binarization has demonstrated promising
efficiency potential on edge devices despite the limited hardware support. The
results and analysis also lead to a promising paradigm for accurate and
efficient binarization. We believe that BiBench will contribute to the broader
adoption of binarization and serve as a foundation for future research. The
code for our BiBench is released https://github.com/htqin/BiBench .
