  While humans excel at continual learning (CL), deep neural networks (DNNs)
exhibit catastrophic forgetting. A salient feature of the brain that allows
effective CL is that it utilizes multiple modalities for learning and
inference, which is underexplored in DNNs. Therefore, we study the role and
interactions of multiple modalities in mitigating forgetting and introduce a
benchmark for multimodal continual learning. Our findings demonstrate that
leveraging multiple views and complementary information from multiple
modalities enables the model to learn more accurate and robust representations.
This makes the model less vulnerable to modality-specific regularities and
considerably mitigates forgetting. Furthermore, we observe that individual
modalities exhibit varying degrees of robustness to distribution shift.
Finally, we propose a method for integrating and aligning the information from
different modalities by utilizing the relational structural similarities
between the data points in each modality. Our method sets a strong baseline
that enables both single- and multimodal inference. Our study provides a
promising case for further exploring the role of multiple modalities in
enabling CL and provides a standard benchmark for future research.
