  Federated Learning (FL) is a popular algorithm to train machine learning
models on user data constrained to edge devices (for example, mobile phones)
due to privacy concerns. Typically, FL is trained with the assumption that no
part of the user data can be egressed from the edge. However, in many
production settings, specific data-modalities/meta-data are limited to be on
device while others are not. For example, in commercial SLU systems, it is
typically desired to prevent transmission of biometric signals (such as audio
recordings of the input prompt) to the cloud, but egress of locally (i.e. on
the edge device) transcribed text to the cloud may be possible. In this work,
we propose a new algorithm called Partial Federated Learning (PartialFL), where
a machine learning model is trained using data where a subset of data
modalities or their intermediate representations can be made available to the
server. We further restrict our model training by preventing the egress of data
labels to the cloud for better privacy, and instead use a contrastive learning
based model objective. We evaluate our approach on two different multi-modal
datasets and show promising results with our proposed approach.
