  Medical imaging data is inherently heterogeneous across different modalities
and clinical centers, posing unique challenges for developing generalizable
foundation models. Conventional entails training distinct models per dataset or
using a shared encoder with modality-specific decoders. However, these
approaches incur heavy computational overheads and suffer from poor
scalability. To address these limitations, we propose the Medical Multimodal
Mixture of Experts (M$^4$oE) framework, leveraging the SwinUNet architecture.
Specifically, M$^4$oE comprises modality-specific experts; each separately
initialized to learn features encoding domain knowledge. Subsequently, a gating
network is integrated during fine-tuning to modulate each expert's contribution
to the collective predictions dynamically. This enhances model interpretability
and generalization ability while retaining expertise specialization.
Simultaneously, the M$^4$oE architecture amplifies the model's parallel
processing capabilities, and it also ensures the model's adaptation to new
modalities with ease. Experiments across three modalities reveal that M$^4$oE
can achieve 3.45% over STU-Net-L, 5.11% over MED3D, and 11.93% over SAM-Med2D
across the MICCAI FLARE22, AMOS2022, and ATLAS2023 datasets. Moreover, M$^4$oE
showcases a significant reduction in training duration with 7 hours less while
maintaining a parameter count that is only 30% of its compared methods. The
code is available at https://github.com/JefferyJiang-YF/M4oE.
