  Learning holistic computational representations in physical, chemical or
biological systems requires the ability to process information from different
distributions and modalities within the same model. Thus, the demand for
multimodal machine learning models has sharply risen for modalities that go
beyond vision and language, such as sequences, graphs, time series, or tabular
data. While there are many available multimodal fusion and alignment
approaches, most of them require end-to-end training, scale quadratically with
the number of modalities, cannot handle cases of high modality imbalance in the
training set, or are highly topology-specific, making them too restrictive for
many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego),
a modular and general-purpose fusion and model merging framework to turn any
set of encoders into a competitive multimodal model with no or minimal
fine-tuning. We achieve this by introducing a wrapper for unimodal encoders
that enforces lightweight dimensionality assumptions between modalities and
harmonises their representations by learning features in the frequency domain
to enable model merging with little signal interference. We show that MM-Lego
1) can be used as a model merging method which achieves competitive performance
with end-to-end fusion models without any fine-tuning, 2) can operate on any
unimodal encoder, and 3) is a model fusion method that, with minimal
fine-tuning, achieves state-of-the-art results on six benchmarked multimodal
biomedical tasks.
