  Effective spatiotemporal feature representation is crucial to the video-based
action recognition task. Focusing on discriminate spatiotemporal feature
learning, we propose Information Fused Temporal Transformation Network (IF-TTN)
for action recognition on top of popular Temporal Segment Network (TSN)
framework. In the network, Information Fusion Module (IFM) is designed to fuse
the appearance and motion features at multiple ConvNet levels for each video
snippet, forming a short-term video descriptor. With fused features as inputs,
Temporal Transformation Networks (TTN) are employed to model middle-term
temporal transformation between the neighboring snippets following a sequential
order. As TSN itself depicts long-term temporal structure by segmental
consensus, the proposed network comprehensively considers multiple granularity
temporal features. Our IF-TTN achieves the state-of-the-art results on two most
popular action recognition datasets: UCF101 and HMDB51. Empirical investigation
reveals that our architecture is robust to the input motion map quality.
Replacing optical flow with the motion vectors from compressed video stream,
the performance is still comparable to the flow-based methods while the testing
speed is 10x faster.
