  This paper investigates the feasibility of federated representation learning
under the constraints of communication cost and privacy protection. Existing
works either conduct annotation-guided local training which requires frequent
communication or aggregates the client models via weight averaging which has
potential risks of privacy exposure. To tackle the above problems, we first
identify that self-supervised contrastive local training is robust against the
non-identically distributed data, which provides the feasibility of longer
local training and thus reduces the communication cost. Then based on the
aforementioned robustness, we propose a novel Federated representation Learning
framework with Ensemble Similarity Distillation~(FLESD) that utilizes this
robustness. At each round of communication, the server first gathers a fraction
of the clients' inferred similarity matrices on a public dataset. Then it
ensembles the similarity matrices and train the global model via similarity
distillation. We verify the effectiveness of FLESD by a series of empirical
experiments and show that, despite stricter constraints, it achieves comparable
results under multiple settings on multiple datasets.
