  Sensing technology is widely used for comprehending the physical world, with
numerous modalities explored in past decades. While there has been considerable
work on multi-modality learning, they all require data of all modalities be
paired. How to leverage multi-modality data with partially pairings remains an
open problem. To tackle this challenge, we introduce the Babel framework,
encompassing the neural network architecture, data preparation and processing,
as well as the training strategies. Babel serves as a scalable pre-trained
multi-modal sensing neural network, currently aligning six sensing modalities,
namely Wi-Fi, mmWave, IMU, LiDAR, video, and depth. To overcome the scarcity of
complete paired data, the key idea of Babel involves transforming the
N-modality alignment into a series of two-modality alignments by devising the
expandable network architecture. This concept is also realized via a series of
novel techniques, including the pre-trained modality tower that capitalizes on
available single-modal networks, and the adaptive training strategy balancing
the contribution of the newly incorporated modality with the previously
established modality alignment.
  Evaluation demonstrates Babel's outstanding performance on eight human
activity recognition datasets, compared to various baselines e.g., the top
multi-modal sensing framework, single-modal sensing networks, and multi-modal
large language models. Babel not only effectively fuses multiple available
modalities (up to 22% accuracy increase), but also enhance the performance of
individual modality (12% averaged accuracy improvement). Case studies also
highlight exciting application scenarios empowered by Babel, including
cross-modality retrieval (i.e., sensing imaging), and bridging LLM for sensing
comprehension.
