  Quantization is commonly used to compress and accelerate deep neural
networks. Quantization assigning the same bit-width to all layers leads to
large accuracy degradation at low precision and is wasteful at high precision
settings. Mixed-precision quantization (MPQ) assigns varied bit-widths to
layers to optimize the accuracy-efficiency trade-off. Existing methods simplify
the MPQ problem by assuming that quantization errors at different layers act
independently. We show that this assumption does not reflect the true behavior
of quantized deep neural networks. We propose the first MPQ algorithm that
captures the cross-layer dependency of quantization error. Our algorithm
(CLADO) enables a fast approximation of pairwise cross-layer error terms by
solving linear equations that require only forward evaluations of the network
on a small amount of data. Decisions on layerwise bit-width assignments are
then determined by optimizing a new MPQ formulation dependent on these
cross-layer quantization errors via the Integer Quadratic Program (IQP), which
can be solved within seconds. We conduct experiments on multiple networks on
the Imagenet dataset and demonstrate an improvement, in top-1 classification
accuracy, of up to 27% over uniform precision quantization, and up to 15% over
existing MPQ methods.
