  Generating natural language descriptions for in-the-wild videos is a
challenging task. Most state-of-the-art methods for solving this problem borrow
existing deep convolutional neural network (CNN) architectures (AlexNet,
GoogLeNet) to extract a visual representation of the input video. However,
these deep CNN architectures are designed for single-label centered-positioned
object classification. While they generate strong semantic features, they have
no inherent structure allowing them to detect multiple objects of different
sizes and locations in the frame. Our paper tries to solve this problem by
integrating the base CNN into several fully convolutional neural networks
(FCNs) to form a multi-scale network that handles multiple receptive field
sizes in the original image. FCNs, previously applied to image segmentation,
can generate class heat-maps efficiently compared to sliding window mechanisms,
and can easily handle multiple scales. To further handle the ambiguity over
multiple objects and locations, we incorporate the Multiple Instance Learning
mechanism (MIL) to consider objects in different positions and at different
scales simultaneously. We integrate our multi-scale multi-instance architecture
with a sequence-to-sequence recurrent neural network to generate sentence
descriptions based on the visual representation. Ours is the first end-to-end
trainable architecture that is capable of multi-scale region processing.
Evaluation on a Youtube video dataset shows the advantage of our approach
compared to the original single-scale whole frame CNN model. Our flexible and
efficient architecture can potentially be extended to support other video
processing tasks.
