  Post-training quantization (PTQ) is widely regarded as one of the most
efficient compression methods practically, benefitting from its data privacy
and low computation costs. We argue that an overlooked problem of oscillation
is in the PTQ methods. In this paper, we take the initiative to explore and
present a theoretical proof to explain why such a problem is essential in PTQ.
And then, we try to solve this problem by introducing a principled and
generalized framework theoretically. In particular, we first formulate the
oscillation in PTQ and prove the problem is caused by the difference in module
capacity. To this end, we define the module capacity (ModCap) under
data-dependent and data-free scenarios, where the differentials between
adjacent modules are used to measure the degree of oscillation. The problem is
then solved by selecting top-k differentials, in which the corresponding
modules are jointly optimized and quantized. Extensive experiments demonstrate
that our method successfully reduces the performance drop and is generalized to
different neural networks and PTQ methods. For example, with 2/4 bit ResNet-50
quantization, our method surpasses the previous state-of-the-art method by
1.9%. It becomes more significant on small model quantization, e.g. surpasses
BRECQ method by 6.61% on MobileNetV2*0.5.
