  Decentralized learning enables a group of collaborative agents to learn
models using a distributed dataset without the need for a central parameter
server. Recently, decentralized learning algorithms have demonstrated
state-of-the-art results on benchmark data sets, comparable with centralized
algorithms. However, the key assumption to achieve competitive performance is
that the data is independently and identically distributed (IID) among the
agents which, in real-life applications, is often not applicable. Inspired by
ideas from continual learning, we propose Cross-Gradient Aggregation (CGA), a
novel decentralized learning algorithm where (i) each agent aggregates
cross-gradient information, i.e., derivatives of its model with respect to its
neighbors' datasets, and (ii) updates its model using a projected gradient
based on quadratic programming (QP). We theoretically analyze the convergence
characteristics of CGA and demonstrate its efficiency on non-IID data
distributions sampled from the MNIST and CIFAR-10 datasets. Our empirical
comparisons show superior learning performance of CGA over existing
state-of-the-art decentralized learning algorithms, as well as maintaining the
improved performance under information compression to reduce peer-to-peer
communication overhead. The code is available here on GitHub.
