  The task of open-set domain generalization (OSDG) involves recognizing novel
classes within unseen domains, which becomes more challenging with multiple
modalities as input. Existing works have only addressed unimodal OSDG within
the meta-learning framework, without considering multimodal scenarios. In this
work, we introduce a novel approach to address Multimodal Open-Set Domain
Generalization (MM-OSDG) for the first time, utilizing self-supervision. To
this end, we introduce two innovative multimodal self-supervised pretext tasks:
Masked Cross-modal Translation and Multimodal Jigsaw Puzzles. These tasks
facilitate the learning of multimodal representative features, thereby
enhancing generalization and open-class detection capabilities. Additionally,
we propose a novel entropy weighting mechanism to balance the loss across
different modalities. Furthermore, we extend our approach to tackle also the
Multimodal Open-Set Domain Adaptation (MM-OSDA) problem, especially in
scenarios where unlabeled data from the target domain is available. Extensive
experiments conducted under MM-OSDG, MM-OSDA, and Multimodal Closed-Set DG
settings on the EPIC-Kitchens and HAC datasets demonstrate the efficacy and
versatility of the proposed approach. Our source code is available at
https://github.com/donghao51/MOOSA.
