  Multimodal fusion can make semantic segmentation more robust. However, fusing
an arbitrary number of modalities remains underexplored. To delve into this
problem, we create the DeLiVER arbitrary-modal segmentation benchmark, covering
Depth, LiDAR, multiple Views, Events, and RGB. Aside from this, we provide this
dataset in four severe weather conditions as well as five sensor failure cases
to exploit modal complementarity and resolve partial outages. To make this
possible, we present the arbitrary cross-modal segmentation model CMNeXt. It
encompasses a Self-Query Hub (SQ-Hub) designed to extract effective information
from any modality for subsequent fusion with the RGB representation and adds
only negligible amounts of parameters (~0.01M) per additional modality. On top,
to efficiently and flexibly harvest discriminative cues from the auxiliary
modalities, we introduce the simple Parallel Pooling Mixer (PPX). With
extensive experiments on a total of six benchmarks, our CMNeXt achieves
state-of-the-art performance on the DeLiVER, KITTI-360, MFNet, NYU Depth V2,
UrbanLF, and MCubeS datasets, allowing to scale from 1 to 81 modalities. On the
freshly collected DeLiVER, the quad-modal CMNeXt reaches up to 66.30% in mIoU
with a +9.10% gain as compared to the mono-modal baseline. The DeLiVER dataset
and our code are at: https://jamycheung.github.io/DELIVER.html.
