  Mixed-precision quantization (MPQ) suffers from the time-consuming process of
searching the optimal bit-width allocation i.e., the policy) for each layer,
especially when using large-scale datasets such as ISLVRC-2012. This limits the
practicality of MPQ in real-world deployment scenarios. To address this issue,
this paper proposes a novel method for efficiently searching for effective MPQ
policies using a small proxy dataset instead of the large-scale dataset used
for training the model. Deviating from the established norm of employing a
consistent dataset for both model training and MPQ policy search stages, our
approach, therefore, yields a substantial enhancement in the efficiency of MPQ
exploration. Nonetheless, using discrepant datasets poses challenges in
searching for a transferable MPQ policy. Driven by the observation that
quantization noise of sub-optimal policy exerts a detrimental influence on the
discriminability of feature representations -- manifesting as diminished class
margins and ambiguous decision boundaries -- our method aims to identify
policies that uphold the discriminative nature of feature representations,
i.e., intra-class compactness and inter-class separation. This general and
dataset-independent property makes us search for the MPQ policy over a rather
small-scale proxy dataset and then the policy can be directly used to quantize
the model trained on a large-scale dataset. Our method offers several
advantages, including high proxy data utilization, no excessive hyper-parameter
tuning, and high searching efficiency. We search high-quality MPQ policies with
the proxy dataset that has only 4% of the data scale compared to the
large-scale target dataset, achieving the same accuracy as searching directly
on the latter, improving MPQ searching efficiency by up to 300 times.
