  Video Large Language Models (Video-LLMs) are flourishing and has advanced
many video-language tasks. As a golden testbed, Video Question Answering
(VideoQA) plays pivotal role in Video-LLM developing. This work conducts a
timely and comprehensive study of Video-LLMs' behavior in VideoQA, aiming to
elucidate their success and failure modes, and provide insights towards more
human-like video understanding and question answering. Our analyses demonstrate
that Video-LLMs excel in VideoQA; they can correlate contextual cues and
generate plausible responses to questions about varied video contents. However,
models falter in handling video temporality, both in reasoning about temporal
content ordering and grounding QA-relevant temporal moments. Moreover, the
models behave unintuitively - they are unresponsive to adversarial video
perturbations while being sensitive to simple variations of candidate answers
and questions. Also, they do not necessarily generalize better. The findings
demonstrate Video-LLMs' QA capability in standard condition yet highlight their
severe deficiency in robustness and interpretability, suggesting the urgent
need on rationales in Video-LLM developing.
