  An ideal description for a given video should fix its gaze on salient and
representative content, which is capable of distinguishing this video from
others. However, the distribution of different words is unbalanced in video
captioning datasets, where distinctive words for describing video-specific
salient objects are far less than common words such as 'a' 'the' and 'person'.
The dataset bias often results in recognition error or detail deficiency of
salient but unusual objects. To address this issue, we propose a novel learning
strategy called Information Loss, which focuses on the relationship between the
video-specific visual content and corresponding representative words. Moreover,
a framework with hierarchical visual representations and an optimized
hierarchical attention mechanism is established to capture the most salient
spatial-temporal visual information, which fully exploits the potential
strength of the proposed learning strategy. Extensive experiments demonstrate
that the ingenious guidance strategy together with the optimized architecture
outperforms state-of-the-art video captioning methods on MSVD with CIDEr score
87.5, and achieves superior CIDEr score 47.7 on MSR-VTT. We also show that our
Information Loss is generic which improves various models by significant
margins.
