  Video Question Answering is a challenging task, which requires the model to
reason over multiple frames and understand the interaction between different
objects to answer questions based on the context provided within the video,
especially in datasets like NExT-QA (Xiao et al., 2021a) which emphasize on
causal and temporal questions. Previous approaches leverage either sub-sampled
information or causal intervention techniques along with complete video
features to tackle the NExT-QA task. In this work we elicit the limitations of
these approaches and propose solutions along four novel directions of
improvements on theNExT-QA dataset. Our approaches attempts to compensate for
the shortcomings in the previous works by systematically attacking each of
these problems by smartly sampling frames, explicitly encoding actions and
creating interventions that challenge the understanding of the model. Overall,
for both single-frame (+6.3%) and complete-video (+1.1%) based approaches, we
obtain the state-of-the-art results on NExT-QA dataset.
