  Multimodal federated learning (FL) aims to enrich model training in FL
settings where devices are collecting measurements across multiple modalities
(e.g., sensors measuring pressure, motion, and other types of data). However,
key challenges to multimodal FL remain unaddressed, particularly in
heterogeneous network settings: (i) the set of modalities collected by each
device will be diverse, and (ii) communication limitations prevent devices from
uploading all their locally trained modality models to the server. In this
paper, we propose Federated Multimodal Fusion learning with Selective modality
communication (FedMFS), a new multimodal fusion FL methodology that can tackle
the above mentioned challenges. The key idea is the introduction of a modality
selection criterion for each device, which weighs (i) the impact of the
modality, gauged by Shapley value analysis, against (ii) the modality model
size as a gauge for communication overhead. This enables FedMFS to flexibly
balance performance against communication costs, depending on resource
constraints and application requirements. Experiments on the real-world
ActionSense dataset demonstrate the ability of FedMFS to achieve comparable
accuracy to several baselines while reducing the communication overhead by over
4x.
