  Neural network quantization enables the deployment of large models on
resource-constrained devices. Current post-training quantization methods fall
short in terms of accuracy for INT4 (or lower) but provide reasonable accuracy
for INT8 (or above). In this work, we study the effect of quantization on the
structure of the loss landscape. Additionally, we show that the structure is
flat and separable for mild quantization, enabling straightforward
post-training quantization methods to achieve good results. We show that with
more aggressive quantization, the loss landscape becomes highly non-separable
with steep curvature, making the selection of quantization parameters more
challenging. Armed with this understanding, we design a method that quantizes
the layer parameters jointly, enabling significant accuracy improvement over
current post-training quantization methods. Reference implementation is
available at
https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq
