  Multimodal AI has demonstrated superior performance over unimodal approaches
by leveraging diverse data sources for more comprehensive analysis. However,
applying this effectiveness in healthcare is challenging due to the limited
availability of public datasets. Federated learning presents an exciting
solution, allowing the use of extensive databases from hospitals and health
centers without centralizing sensitive data, thus maintaining privacy and
security. Yet, research in multimodal federated learning, particularly in
scenarios with missing modalities a common issue in healthcare datasets remains
scarce, highlighting a critical area for future exploration. Toward this, we
propose a novel method for multimodal federated learning with missing
modalities. Our contribution lies in a novel cross-modal data augmentation by
retrieval, leveraging the small publicly available dataset to fill the missing
modalities in the clients. Our method learns the parameters in a federated
manner, ensuring privacy protection and improving performance in multiple
challenging multimodal benchmarks in the medical domain, surpassing several
competitive baselines. Code Available: https://github.com/bhattarailab/CAR-MFL
