  Automatically describing video content with natural language has been
attracting much attention in CV and NLP communities. Most existing methods
predict one word at a time, and by feeding the last generated word back as
input at the next time, while the other generated words are not fully
exploited. Furthermore, traditional methods optimize the model using all the
training samples in each epoch without considering their learning situations,
which leads to a lot of unnecessary training and can not target the difficult
samples. To address these issues, we propose a text-based dynamic attention
model named TDAM, which imposes a dynamic attention mechanism on all the
generated words with the motivation to improve the context semantic information
and enhance the overall control of the whole sentence. Moreover, the text-based
dynamic attention mechanism and the visual attention mechanism are linked
together to focus on the important words. They can benefit from each other
during training. Accordingly, the model is trained through two steps: "starting
from scratch" and "checking for gaps". The former uses all the samples to
optimize the model, while the latter only trains for samples with poor control.
Experimental results on the popular datasets MSVD and MSR-VTT demonstrate that
our non-ensemble model outperforms the state-of-the-art video captioning
benchmarks.
