  The impact of quantization on the overall performance of deep learning models
is a well-studied problem. However, understanding and mitigating its effects on
a more fine-grained level is still lacking, especially for harder tasks such as
object detection with both classification and regression objectives. This work
defines the performance for a subset of task-critical categories, i.e. the
critical-category performance, as a crucial yet largely overlooked fine-grained
objective for detection tasks. We analyze the impact of quantization at the
category-level granularity, and propose methods to improve performance for the
critical categories. Specifically, we find that certain critical categories
have a higher sensitivity to quantization, and are prone to overfitting after
quantization-aware training (QAT). To explain this, we provide theoretical and
empirical links between their performance gaps and the corresponding loss
landscapes with the Fisher information framework. Using this evidence, we apply
a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace
regularization for the QAT on the critical-category loss landscape. The
proposed methods improve critical-category metrics of the quantized
transformer-based DETR detectors. They are even more significant in case of
larger models and higher number of classes where the overfitting becomes more
severe. For example, our methods lead to 10.4% and 14.5% mAP gains for,
correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted
critical classes in the COCO Panoptic dataset.
