  While the Self-Attention mechanism in the Transformer model has proven to be
effective in many domains, we observe that it is less effective in more diverse
settings (e.g. multimodality) due to the varying granularity of each token and
the high computational demands of lengthy sequences. To address the challenges,
we introduce the Learnable Attention Mask (LAM), strategically designed to
globally regulate attention maps and prioritize critical tokens within the
sequence. Leveraging the Self-Attention module in a BERT-like transformer
network, our approach adeptly captures associations between tokens. The
extension of the LAM to a multi-layer version accommodates the varied
information aspects embedded at each layer of the Transformer network.
Comprehensive experimental validation on various datasets, such as MADv2,
QVHighlights, ImageNet 1K, and MSRVTT, demonstrates the efficacy of the LAM,
exemplifying its ability to enhance model performance while mitigating
redundant computations. This pioneering approach presents a significant
advancement in enhancing the understanding of complex scenarios, such as in
movie understanding.
