  This paper proposes a Video Graph Transformer (VGT) model for Video Quetion
Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic
graph transformer module which encodes video by explicitly capturing the visual
objects, their relations, and dynamics for complex spatio-temporal reasoning;
and 2) it exploits disentangled video and text Transformers for relevance
comparison between the video and text to perform QA, instead of entangled
cross-modal Transformer for answer classification. Vision-text communication is
done by additional cross-modal interaction modules. With more reasonable video
encoding and QA solution, we show that VGT can achieve much better performances
on VideoQA tasks that challenge dynamic relation reasoning than prior arts in
the pretraining-free scenario. Its performances even surpass those models that
are pretrained with millions of external data. We further show that VGT can
also benefit a lot from self-supervised cross-modal pretraining, yet with
orders of magnitude smaller data. These results clearly demonstrate the
effectiveness and superiority of VGT, and reveal its potential for more
data-efficient pretraining. With comprehensive analyses and some heuristic
observations, we hope that VGT can promote VQA research beyond coarse
recognition/description towards fine-grained relation reasoning in realistic
videos. Our code is available at https://github.com/sail-sg/VGT.
