  This paper studies deep network architectures to address the problem of video
classification. A multi-stream framework is proposed to fully utilize the rich
multimodal information in videos. Specifically, we first train three
Convolutional Neural Networks to model spatial, short-term motion and audio
clues respectively. Long Short Term Memory networks are then adopted to explore
long-term temporal dynamics. With the outputs of the individual streams, we
propose a simple and effective fusion method to generate the final predictions,
where the optimal fusion weights are learned adaptively for each class, and the
learning process is regularized by automatically estimated class relationships.
Our contributions are two-fold. First, the proposed multi-stream framework is
able to exploit multimodal features that are more comprehensive than those
previously attempted. Second, we demonstrate that the adaptive fusion method
using the class relationship as a regularizer outperforms traditional
alternatives that estimate the weights in a "free" fashion. Our framework
produces significantly better results than the state of the arts on two popular
benchmarks, 92.2\% on UCF-101 (without using audio) and 84.9\% on Columbia
Consumer Videos.
