  Recent researches on information bottleneck shed new light on the continuous
attempts to open the black box of neural signal encoding. Inspired by the
problem of lossy signal compression for wireless communication, this paper
presents a Bitwise Information Bottleneck approach for quantizing and encoding
neural network activations. Based on the rate-distortion theory, the Bitwise
Information Bottleneck attempts to determine the most significant bits in
activation representation by assigning and approximating the sparse coefficient
associated with each bit. Given the constraint of a limited average code rate,
the information bottleneck minimizes the rate-distortion for optimal activation
quantization in a flexible layer-by-layer manner. Experiments over ImageNet and
other datasets show that, by minimizing the quantization rate-distortion of
each layer, the neural network with information bottlenecks achieves the
state-of-the-art accuracy with low-precision activation. Meanwhile, by reducing
the code rate, the proposed method can improve the memory and computational
efficiency by over six times compared with the deep neural network with
standard single-precision representation. Codes will be available on GitHub
when the paper is accepted \url{https://github.com/BitBottleneck/PublicCode}.
