  The increasing size of deep neural networks necessitates effective model
compression to improve computational efficiency and reduce their memory
footprint. Sparsity and quantization are two prominent compression methods that
have individually demonstrated significant reduction in computational and
memory footprints while preserving model accuracy. While effective, the
interplay between these two methods remains an open question. In this paper, we
investigate the interaction between these two methods and assess whether their
combination impacts final model accuracy. We mathematically prove that applying
sparsity before quantization is the optimal sequence for these operations,
minimizing error in computation. Our empirical studies across a wide range of
models, including OPT and Llama model families (125M-8B) and ViT corroborate
these theoretical findings. In addition, through rigorous analysis, we
demonstrate that sparsity and quantization are not orthogonal; their
interaction can significantly harm model accuracy, with quantization error
playing a dominant role in this degradation. Our findings extend to the
efficient deployment of large models in resource-limited compute platforms and
reduce serving cost, offering insights into best practices for applying these
compression methods to maximize efficacy without compromising accuracy.
