  Deep learning models have enjoyed great success for image related computer
vision tasks like image classification and object detection. For video related
tasks like human action recognition, however, the advancements are not as
significant yet. The main challenge is the lack of effective and efficient
models in modeling the rich temporal spatial information in a video. We
introduce a simple yet effective operation, termed Temporal-Spatial Mapping
(TSM), for capturing the temporal evolution of the frames by jointly analyzing
all the frames of a video. We propose a video level 2D feature representation
by transforming the convolutional features of all frames to a 2D feature map,
referred to as VideoMap. With each row being the vectorized feature
representation of a frame, the temporal-spatial features are compactly
represented, while the temporal dynamic evolution is also well embedded. Based
on the VideoMap representation, we further propose a temporal attention model
within a shallow convolutional neural network to efficiently exploit the
temporal-spatial dynamics. The experiment results show that the proposed scheme
achieves the state-of-the-art performance, with 4.2% accuracy gain over
Temporal Segment Network (TSN), a competing baseline method, on the challenging
human action benchmark dataset HMDB51.
