  The use of machine learning (ML) for cancer staging through medical image
analysis has gained substantial interest across medical disciplines. When
accompanied by the innovative federated learning (FL) framework, ML techniques
can further overcome privacy concerns related to patient data exposure. Given
the frequent presence of diverse data modalities within patient records,
leveraging FL in a multi-modal learning framework holds considerable promise
for cancer staging.
  However, existing works on multi-modal FL often presume that all
data-collecting institutions have access to all data modalities. This
oversimplified approach neglects institutions that have access to only a
portion of data modalities within the system. In this work, we introduce a
novel FL architecture designed to accommodate not only the heterogeneity of
data samples, but also the inherent heterogeneity/non-uniformity of data
modalities across institutions. We shed light on the challenges associated with
varying convergence speeds observed across different data modalities within our
FL system. Subsequently, we propose a solution to tackle these challenges by
devising a distributed gradient blending and proximity-aware client weighting
strategy tailored for multi-modal FL. To show the superiority of our method, we
conduct experiments using The Cancer Genome Atlas program (TCGA) datalake
considering different cancer types and three modalities of data: mRNA
sequences, histopathological image data, and clinical information. Our results
further unveil the impact and severity of class-based vs type-based
heterogeneity across institutions on the model performance, which widens the
perspective to the notion of data heterogeneity in multi-modal FL literature.
