  Two-stream architecture have shown strong performance in video classification
task. The key idea is to learn spatio-temporal features by fusing convolutional
networks spatially and temporally. However, there are some problems within such
architecture. First, it relies on optical flow to model temporal information,
which are often expensive to compute and store. Second, it has limited ability
to capture details and local context information for video data. Third, it
lacks explicit semantic guidance that greatly decrease the classification
performance. In this paper, we proposed a new two-stream based deep framework
for video classification to discover spatial and temporal information only from
RGB frames, moreover, the multi-scale pyramid attention (MPA) layer and the
semantic adversarial learning (SAL) module is introduced and integrated in our
framework. The MPA enables the network capturing global and local feature to
generate a comprehensive representation for video, and the SAL can make this
representation gradually approximate to the real video semantics in an
adversarial manner. Experimental results on two public benchmarks demonstrate
our proposed methods achieves state-of-the-art results on standard video
datasets.
