  Network quantization has rapidly become one of the most widely used methods
to compress and accelerate deep neural networks. Recent efforts propose to
quantize weights and activations from different layers with different precision
to improve the overall performance. However, it is challenging to find the
optimal bitwidth (i.e., precision) for weights and activations of each layer
efficiently. Meanwhile, it is yet unclear how to perform convolution for
weights and activations of different precision efficiently on generic hardware
platforms. To resolve these two issues, in this paper, we first propose an
Efficient Bitwidth Search (EBS) algorithm, which reuses the meta weights for
different quantization bitwidth and thus the strength for each candidate
precision can be optimized directly w.r.t the objective without superfluous
copies, reducing both the memory and computational cost significantly. Second,
we propose a binary decomposition algorithm that converts weights and
activations of different precision into binary matrices to make the mixed
precision convolution efficient and practical. Experiment results on CIFAR10
and ImageNet datasets demonstrate our mixed precision QNN outperforms the
handcrafted uniform bitwidth counterparts and other mixed precision techniques.
