  With the increasing amount of multimedia data on modern mobile systems and
IoT infrastructures, harnessing these rich multimodal data without breaching
user privacy becomes a critical issue. Federated learning (FL) serves as a
privacy-conscious alternative to centralized machine learning. However,
existing FL methods extended to multimodal data all rely on model aggregation
on single modality level, which restrains the server and clients to have
identical model architecture for each modality. This limits the global model in
terms of both model complexity and data capacity, not to mention task
diversity. In this work, we propose Contrastive Representation Ensemble and
Aggregation for Multimodal FL (CreamFL), a multimodal federated learning
framework that enables training larger server models from clients with
heterogeneous model architectures and data modalities, while only communicating
knowledge on public dataset. To achieve better multimodal representation
fusion, we design a global-local cross-modal ensemble strategy to aggregate
client representations. To mitigate local model drift caused by two
unprecedented heterogeneous factors stemming from multimodal discrepancy
(modality gap and task gap), we further propose two inter-modal and intra-modal
contrasts to regularize local training, which complements information of the
absent modality for uni-modal clients and regularizes local clients to head
towards global consensus. Thorough evaluations and ablation studies on
image-text retrieval and visual question answering tasks showcase the
superiority of CreamFL over state-of-the-art FL methods and its practical
value.
