  We analyze the effect of quantizing weights and activations of neural
networks on their loss and derive a simple regularization scheme that improves
robustness against post-training quantization. By training quantization-ready
networks, our approach enables storing a single set of weights that can be
quantized on-demand to different bit-widths as energy and memory requirements
of the application change. Unlike quantization-aware training using the
straight-through estimator that only targets a specific bit-width and requires
access to training data and pipeline, our regularization-based method paves the
way for "on the fly'' post-training quantization to various bit-widths. We show
that by modeling quantization as a $\ell_\infty$-bounded perturbation, the
first-order term in the loss expansion can be regularized using the
$\ell_1$-norm of gradients. We experimentally validate the effectiveness of our
regularization scheme on different architectures on CIFAR-10 and ImageNet
datasets.
