  Recent research highlights frequent model communication as a significant
bottleneck to the efficiency of decentralized machine learning (ML), especially
for large-scale and over-parameterized neural networks (NNs). To address this,
we present Malcom-PSGD, a novel decentralized ML algorithm that combines
gradient compression techniques with model sparsification. We promote model
sparsity by adding $\ell_1$ regularization to the objective and present a
decentralized proximal SGD method for training. Our approach employs vector
source coding and dithering-based quantization for the compressed gradient
communication of sparsified models. Our analysis demonstrates that Malcom-PSGD
achieves a convergence rate of $\mathcal{O}(1/\sqrt{t})$ with respect to the
iterations $t$, assuming a constant consensus and learning rate. This result is
supported by our proof for the convergence of non-convex compressed Proximal
SGD methods. Additionally, we conduct a bit analysis, providing a closed-form
expression for the communication costs associated with Malcom-PSGD. Numerical
results verify our theoretical findings and demonstrate that our method reduces
communication costs by approximately $75\%$ when compared to the
state-of-the-art.
