  Driver action recognition has significantly advanced in enhancing
driver-vehicle interactions and ensuring driving safety by integrating multiple
modalities, such as infrared and depth. Nevertheless, compared to RGB modality
only, it is always laborious and costly to collect extensive data for all types
of non-RGB modalities in car cabin environments. Therefore, previous works have
suggested independently learning each non-RGB modality by fine-tuning a model
pre-trained on RGB videos, but these methods are less effective in extracting
informative features when faced with newly-incoming modalities due to large
domain gaps. In contrast, we propose a Continual Cross-Modal Mapping Network
(CM2-Net) to continually learn each newly-incoming modality with instructive
prompts from the previously-learned modalities. Specifically, we have developed
Accumulative Cross-modal Mapping Prompting (ACMP), to map the discriminative
and informative features learned from previous modalities into the feature
space of newly-incoming modalities. Then, when faced with newly-incoming
modalities, these mapped features are able to provide effective prompts for
which features should be extracted and prioritized. These prompts are
accumulating throughout the continual learning process, thereby boosting
further recognition performances. Extensive experiments conducted on the
Drive&Act dataset demonstrate the performance superiority of CM2-Net on both
uni- and multi-modal driver action recognition.
