  We introduce NExT-QA, a rigorously designed video question answering
(VideoQA) benchmark to advance video understanding from describing to
explaining the temporal actions. Based on the dataset, we set up multi-choice
and open-ended QA tasks targeting causal action reasoning, temporal action
reasoning, and common scene comprehension. Through extensive analysis of
baselines and established VideoQA techniques, we find that top-performing
methods excel at shallow scene descriptions but are weak in causal and temporal
action reasoning. Furthermore, the models that are effective on multi-choice
QA, when adapted to open-ended QA, still struggle in generalizing the answers.
This raises doubt on the ability of these models to reason and highlights
possibilities for improvement. With detailed results for different question
types and heuristic observations for future works, we hope NExT-QA will guide
the next generation of VQA research to go beyond superficial scene description
towards a deeper understanding of videos. (The dataset and related resources
are available at https://github.com/doc-doc/NExT-QA.git)
