  Video captioning aims to describe events in a video with natural language. In
recent years, many works have focused on improving captioning models'
performance. However, like other text generation tasks, it risks introducing
factual errors not supported by the input video. These factual errors can
seriously affect the quality of the generated text, sometimes making it
completely unusable. Although factual consistency has received much research
attention in text-to-text tasks (e.g., summarization), it is less studied in
the context of vision-based text generation. In this work, we conduct a
detailed human evaluation of the factuality in video captioning and collect two
annotated factuality datasets. We find that 57.0% of the model-generated
sentences have factual errors, indicating it is a severe problem in this field.
However, existing evaluation metrics are mainly based on n-gram matching and
show little correlation with human factuality annotation. We further propose a
weakly-supervised, model-based factuality metric FactVC, which outperforms
previous metrics on factuality evaluation of video captioning. The datasets and
metrics will be released to promote future research for video captioning.
