  Due to the notorious modality imbalance problem, multimodal learning (MML)
leads to the phenomenon of optimization imbalance, thus struggling to achieve
satisfactory performance. Recently, some representative methods have been
proposed to boost the performance, mainly focusing on adaptive adjusting the
optimization of each modality to rebalance the learning speed of dominant and
non-dominant modalities. To better facilitate the interaction of model
information in multimodal learning, in this paper, we propose a novel
multimodal learning method, called modal-aware interactive enhancement (MIE).
Specifically, we first utilize an optimization strategy based on sharpness
aware minimization (SAM) to smooth the learning objective during the forward
phase. Then, with the help of the geometry property of SAM, we propose a
gradient modification strategy to impose the influence between different
modalities during the backward phase. Therefore, we can improve the
generalization ability and alleviate the modality forgetting phenomenon
simultaneously for multimodal learning. Extensive experiments on widely used
datasets demonstrate that our proposed method can outperform various
state-of-the-art baselines to achieve the best performance.
