  Decentralized learning with private data is a central problem in machine
learning. We propose a novel distillation-based decentralized learning
technique that allows multiple agents with private non-iid data to learn from
each other, without having to share their data, weights or weight updates. Our
approach is communication efficient, utilizes an unlabeled public dataset and
uses multiple auxiliary heads for each client, greatly improving training
efficiency in the case of heterogeneous data. This approach allows individual
models to preserve and enhance performance on their private tasks while also
dramatically improving their performance on the global aggregated data
distribution. We study the effects of data and model architecture heterogeneity
and the impact of the underlying communication graph topology on learning
efficiency and show that our agents can significantly improve their performance
compared to learning in isolation.
