  Leveraging information across diverse modalities is known to enhance
performance on multimodal segmentation tasks. However, effectively fusing
information from different modalities remains challenging due to the unique
characteristics of each modality. In this paper, we propose a novel fusion
strategy that can effectively fuse information from different modality
combinations. We also propose a new model named Multi-Modal Segmentation
TransFormer (MMSFormer) that incorporates the proposed fusion strategy to
perform multimodal material and semantic segmentation tasks. MMSFormer
outperforms current state-of-the-art models on three different datasets. As we
begin with only one input modality, performance improves progressively as
additional modalities are incorporated, showcasing the effectiveness of the
fusion block in combining useful information from diverse input modalities.
Ablation studies show that different modules in the fusion block are crucial
for overall model performance. Furthermore, our ablation studies also highlight
the capacity of different input modalities to improve performance in the
identification of different types of materials. The code and pretrained models
will be made available at https://github.com/csiplab/MMSFormer.
