  Multimodal Federated Learning (MMFL) utilizes multiple modalities in each
client to build a more powerful Federated Learning (FL) model than its unimodal
counterpart. However, the impact of missing modality in different clients, also
called modality incongruity, has been greatly overlooked. This paper, for the
first time, analyses the impact of modality incongruity and reveals its
connection with data heterogeneity across participating clients. We
particularly inspect whether incongruent MMFL with unimodal and multimodal
clients is more beneficial than unimodal FL. Furthermore, we examine three
potential routes of addressing this issue. Firstly, we study the effectiveness
of various self-attention mechanisms towards incongruity-agnostic information
fusion in MMFL. Secondly, we introduce a modality imputation network (MIN)
pre-trained in a multimodal client for modality translation in unimodal clients
and investigate its potential towards mitigating the missing modality problem.
Thirdly, we assess the capability of client-level and server-level
regularization techniques towards mitigating modality incongruity effects.
Experiments are conducted under several MMFL settings on two publicly available
real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology
reports.
