  Post-training quantization (PTQ) has emerged as a practical approach to
compress large neural networks, making them highly efficient for deployment.
However, effectively reducing these models to their low-bit counterparts
without compromising the original accuracy remains a key challenge. In this
paper, we propose an innovative PTQ algorithm termed COMQ, which sequentially
conducts coordinate-wise minimization of the layer-wise reconstruction errors.
We consider the widely used integer quantization, where every quantized weight
can be decomposed into a shared floating-point scalar and an integer bit-code.
Within a fixed layer, COMQ treats all the scaling factor(s) and bit-codes as
the variables of the reconstruction error. Every iteration improves this error
along a single coordinate while keeping all other variables constant. COMQ is
easy to use and requires no hyper-parameter tuning. It instead involves only
dot products and rounding operations. We update these variables in a carefully
designed greedy order, significantly enhancing the accuracy. COMQ achieves
remarkable results in quantizing 4-bit Vision Transformers, with a negligible
loss of less than 1% in Top-1 accuracy. In 4-bit INT quantization of
convolutional neural networks, COMQ maintains near-lossless accuracy with a
minimal drop of merely 0.3% in Top-1 accuracy.
