  Video advertisement content structuring aims to segment a given video
advertisement and label each segment on various dimensions, such as
presentation form, scene, and style. Different from real-life videos, video
advertisements contain sufficient and useful multi-modal content like caption
and speech, which provides crucial video semantics and would enhance the
structuring process. In this paper, we propose a multi-modal encoder to learn
multi-modal representation from video advertisements by interacting between
video-audio and text. Based on multi-modal representation, we then apply
Boundary-Matching Network to generate temporal proposals. To make the proposals
more accurate, we refine generated proposals by scene-guided alignment and
re-ranking. Finally, we incorporate proposal located embeddings into the
introduced multi-modal encoder to capture temporal relationships between local
features of each proposal and global features of the whole video for
classification. Experimental results show that our method achieves
significantly improvement compared with several baselines and Rank 1 on the
task of Multi-modal Ads Video Understanding in ACM Multimedia 2021 Grand
Challenge. Ablation study further shows that leveraging multi-modal content
like caption and speech in video advertisements significantly improve the
performance.
