  Federated learning (FL) has obtained tremendous progress in providing
collaborative training solutions for distributed data silos with privacy
guarantees. However, few existing works explore a more realistic scenario where
the clients hold multiple data modalities. In this paper, we aim to solve a
novel challenge in multi-modal federated learning (MFL) -- modality missing --
the clients may lose part of the modalities in their local data sets. To tackle
the problems, we propose a novel multi-modal federated learning method,
Federated Multi-modal contrastiVe training with Pre-trained completion
(FedMVP), which integrates the large-scale pre-trained models to enhance the
federated training. In the proposed FedMVP framework, each client deploys a
large-scale pre-trained model with frozen parameters for modality completion
and representation knowledge transfer, enabling efficient and robust local
training. On the server side, we utilize generated data to uniformly measure
the representation similarity among the uploaded client models and construct a
graph perspective to aggregate them according to their importance in the
system. We demonstrate that the model achieves superior performance over two
real-world image-text classification datasets and is robust to the performance
degradation caused by missing modality.
