  This article proposes a communication-efficient decentralized deep learning
algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an
empirical risk, every worker in L-FGADMM periodically communicates with two
neighbors, in which the periods are separately adjusted for different layers of
its deep neural network. A constrained optimization problem for this setting is
formulated and solved using the stochastic version of GADMM proposed in our
prior work. Numerical evaluations show that by less frequently exchanging the
largest layer, L-FGADMM can significantly reduce the communication cost,
without compromising the convergence speed. Surprisingly, despite less
exchanged information and decentralized operations, intermittently skipping the
largest layer consensus in L-FGADMM creates a regularizing effect, thereby
achieving the test accuracy as high as federated learning (FL), a baseline
method with the entire layer consensus by the aid of a central entity.
