  Understanding videos is an important research topic for multimodal learning.
Leveraging large-scale datasets of web-crawled video-text pairs as weak
supervision has become a pre-training paradigm for learning joint
representations and showcased remarkable potential in video understanding
tasks. However, videos can be multi-event and multi-grained, while these
video-text pairs usually contain only broad-level video captions. This raises a
question: with such weak supervision, can video representation in
video-language models gain the ability to distinguish even factual
discrepancies in textual description and understand fine-grained events? To
address this, we introduce SPOT Prober, to benchmark existing video-language
models's capacities of distinguishing event-level discrepancies as an indicator
of models' event understanding ability. Our approach involves extracting events
as tuples (<Subject, Predicate, Object, Attribute, Timestamps>) from videos and
generating false event tuples by manipulating tuple components systematically.
We reevaluate the existing video-language models with these positive and
negative captions and find they fail to distinguish most of the manipulated
events. Based on our findings, we propose to plug in these manipulated event
captions as hard negative samples and find them effective in enhancing models
for event understanding.
