  Post-training quantization (PTQ) is a popular method for compressing deep
neural networks (DNNs) without modifying their original architecture or
training procedures. Despite its effectiveness and convenience, the reliability
of PTQ methods in the presence of some extrem cases such as distribution shift
and data noise remains largely unexplored. This paper first investigates this
problem on various commonly-used PTQ methods. We aim to answer several research
questions related to the influence of calibration set distribution variations,
calibration paradigm selection, and data augmentation or sampling strategies on
PTQ reliability. A systematic evaluation process is conducted across a wide
range of tasks and commonly-used PTQ paradigms. The results show that most
existing PTQ methods are not reliable enough in term of the worst-case group
performance, highlighting the need for more robust methods. Our findings
provide insights for developing PTQ methods that can effectively handle
distribution shift scenarios and enable the deployment of quantized DNNs in
real-world applications.
