  Video Question Answering (VideoQA) has emerged as a challenging frontier in
the field of multimedia processing, requiring intricate interactions between
visual and textual modalities. Simply uniformly sampling frames or
indiscriminately aggregating frame-level visual features often falls short in
capturing the nuanced and relevant contexts of videos to well perform VideoQA.
To mitigate these issues, we propose VidF4, a novel VideoQA framework equipped
with tailored frame selection strategy for effective and efficient VideoQA. We
propose three frame-scoring mechanisms that consider both question relevance
and inter-frame similarity to evaluate the importance of each frame for a given
question on the video. Furthermore, we design a differentiable adaptive frame
sampling mechanism to facilitate end-to-end training for the frame selector and
answer generator. The experimental results across three widely adopted
benchmarks demonstrate that our model consistently outperforms existing VideoQA
methods, establishing a new SOTA across NExT-QA (+0.3%), STAR (+0.9%), and TVQA
(+1.0%). Furthermore, through both quantitative and qualitative analyses, we
validate the effectiveness of each design choice.
