  Surveillance videos are an essential component of daily life with various
critical applications, particularly in public security. However, current
surveillance video tasks mainly focus on classifying and localizing anomalous
events. Existing methods are limited to detecting and classifying the
predefined events with unsatisfactory semantic understanding, although they
have obtained considerable performance. To address this issue, we propose a new
research direction of surveillance video-and-language understanding, and
construct the first multimodal surveillance video dataset. We manually annotate
the real-world surveillance dataset UCF-Crime with fine-grained event content
and timing. Our newly annotated dataset, UCA (UCF-Crime Annotation), contains
23,542 sentences, with an average length of 20 words, and its annotated videos
are as long as 110.7 hours. Furthermore, we benchmark SOTA models for four
multimodal tasks on this newly created dataset, which serve as new baselines
for surveillance video-and-language understanding. Through our experiments, we
find that mainstream models used in previously publicly available datasets
perform poorly on surveillance video, which demonstrates the new challenges in
surveillance video-and-language understanding. To validate the effectiveness of
our UCA, we conducted experiments on multimodal anomaly detection. The results
demonstrate that our multimodal surveillance learning can improve the
performance of conventional anomaly detection tasks. All the experiments
highlight the necessity of constructing this dataset to advance surveillance
AI. The link to our dataset is provided at:
https://xuange923.github.io/Surveillance-Video-Understanding.
