  Multi-modal video question answering aims to predict correct answer and
localize the temporal boundary relevant to the question. The temporal
annotations of questions improve QA performance and interpretability of recent
works, but they are usually empirical and costly. To avoid the temporal
annotations, we devise a weakly supervised question grounding (WSQG) setting,
where only QA annotations are used and the relevant temporal boundaries are
generated according to the temporal attention scores. To substitute the
temporal annotations, we transform the correspondence between frames and
subtitles to Frame-Subtitle (FS) self-supervision, which helps to optimize the
temporal attention scores and hence improve the video-language understanding in
VideoQA model. The extensive experiments on TVQA and TVQA+ datasets demonstrate
that the proposed WSQG strategy gets comparable performance on question
grounding, and the FS self-supervision helps improve the question answering and
grounding performance on both QA-supervision only and full-supervision
settings.
