  We study how to train personalized models for different tasks on
decentralized devices with limited local data. We propose "Structured
Cooperative Learning (SCooL)", in which a cooperation graph across devices is
generated by a graphical model prior to automatically coordinate mutual
learning between devices. By choosing graphical models enforcing different
structures, we can derive a rich class of existing and novel decentralized
learning algorithms via variational inference. In particular, we show three
instantiations of SCooL that adopt Dirac distribution, stochastic block model
(SBM), and attention as the prior generating cooperation graphs. These EM-type
algorithms alternate between updating the cooperation graph and cooperative
learning of local models. They can automatically capture the cross-task
correlations among devices by only monitoring their model updating in order to
optimize the cooperation graph. We evaluate SCooL and compare it with existing
decentralized learning methods on an extensive set of benchmarks, on which
SCooL always achieves the highest accuracy of personalized models and
significantly outperforms other baselines on communication efficiency. Our code
is available at https://github.com/ShuangtongLi/SCooL.
