  Decentralized learning has been advocated and widely deployed to make
efficient use of distributed datasets, with an extensive focus on supervised
learning (SL) problems. Unfortunately, the majority of real-world data are
unlabeled and can be highly heterogeneous across sources. In this work, we
carefully study decentralized learning with unlabeled data through the lens of
self-supervised learning (SSL), specifically contrastive visual representation
learning. We study the effectiveness of a range of contrastive learning
algorithms under decentralized learning settings, on relatively large-scale
datasets including ImageNet-100, MS-COCO, and a new real-world robotic
warehouse dataset. Our experiments show that the decentralized SSL (Dec-SSL)
approach is robust to the heterogeneity of decentralized datasets, and learns
useful representation for object classification, detection, and segmentation
tasks. This robustness makes it possible to significantly reduce communication
and reduce the participation ratio of data sources with only minimal drops in
performance. Interestingly, using the same amount of data, the representation
learned by Dec-SSL can not only perform on par with that learned by centralized
SSL which requires communication and excessive data storage costs, but also
sometimes outperform representations extracted from decentralized SL which
requires extra knowledge about the data labels. Finally, we provide theoretical
insights into understanding why data heterogeneity is less of a concern for
Dec-SSL objectives, and introduce feature alignment and clustering techniques
to develop a new Dec-SSL algorithm that further improves the performance, in
the face of highly non-IID data. Our study presents positive evidence to
embrace unlabeled data in decentralized learning, and we hope to provide new
insights into whether and why decentralized SSL is effective.
