  Video captioning aims to automatically generate natural language descriptions
of video content, which has drawn a lot of attention recent years. Generating
accurate and fine-grained captions needs to not only understand the global
content of video, but also capture the detailed object information. Meanwhile,
video representations have great impact on the quality of generated captions.
Thus, it is important for video captioning to capture salient objects with
their detailed temporal dynamics, and represent them using discriminative
spatio-temporal representations. In this paper, we propose a new video
captioning approach based on object-aware aggregation with bidirectional
temporal graph (OA-BTG), which captures detailed temporal dynamics for salient
objects in video, and learns discriminative spatio-temporal representations by
performing object-aware local feature aggregation on detected object regions.
The main novelties and advantages are: (1) Bidirectional temporal graph: A
bidirectional temporal graph is constructed along and reversely along the
temporal order, which provides complementary ways to capture the temporal
trajectories for each salient object. (2) Object-aware aggregation: Learnable
VLAD (Vector of Locally Aggregated Descriptors) models are constructed on
object temporal trajectories and global frame sequence, which performs
object-aware aggregation to learn discriminative representations. A
hierarchical attention mechanism is also developed to distinguish different
contributions of multiple objects. Experiments on two widely-used datasets
demonstrate our OA-BTG achieves state-of-the-art performance in terms of
BLEU@4, METEOR and CIDEr metrics.
