  Privacy-preserving machine learning has become a key conundrum for
multi-party artificial intelligence. Federated learning (FL) and Split Learning
(SL) are two frameworks that enable collaborative learning while keeping the
data private (on device). In FL, each data holder trains a model locally and
releases it to a central server for aggregation. In SL, the clients must
release individual cut-layer activations (smashed data) to the server and wait
for its response (during both inference and back propagation). While relevant
in several settings, both of these schemes have a high communication cost, rely
on server-level computation algorithms and do not allow for tunable levels of
collaboration. In this work, we present a novel approach for privacy-preserving
machine learning, where the clients collaborate via online knowledge
distillation using a contrastive loss (contrastive w.r.t. the labels). The goal
is to ensure that the participants learn similar features on similar classes
without sharing their input data. To do so, each client releases averaged last
hidden layer activations of similar labels to a central server that only acts
as a relay (i.e., is not involved in the training or aggregation of the
models). Then, the clients download these last layer activations (feature
representations) of the ensemble of users and distill their knowledge in their
personal model using a contrastive objective. For cross-device applications
(i.e., small local datasets and limited computational capacity), this approach
increases the utility of the models compared to independent learning and other
federated knowledge distillation (FD) schemes, is communication efficient and
is scalable with the number of clients. We prove theoretically that our
framework is well-posed, and we benchmark its performance against standard FD
and FL on various datasets using different model architectures.
