  Multi-modal models have shown a promising capability to effectively integrate
information from various sources, yet meanwhile, they are found vulnerable to
pervasive perturbations, such as uni-modal attacks and missing conditions. To
counter these perturbations, robust multi-modal representations are highly
expected, which are positioned well away from the discriminative multi-modal
decision boundary. In this paper, different from conventional empirical
studies, we focus on a commonly used joint multi-modal framework and
theoretically discover that larger uni-modal representation margins and more
reliable integration for modalities are essential components for achieving
higher robustness. This discovery can further explain the limitation of
multi-modal robustness and the phenomenon that multi-modal models are often
vulnerable to attacks on the specific modality. Moreover, our analysis reveals
how the widespread issue, that the model has different preferences for
modalities, limits the multi-modal robustness by influencing the essential
components and could lead to attacks on the specific modality highly effective.
Inspired by our theoretical finding, we introduce a training procedure called
Certifiable Robust Multi-modal Training (CRMT), which can alleviate this
influence from modality preference and explicitly regulate essential components
to significantly improve robustness in a certifiable manner. Our method
demonstrates substantial improvements in performance and robustness compared
with existing methods. Furthermore, our training procedure can be easily
extended to enhance other robust training strategies, highlighting its
credibility and flexibility.
