  The CNN-encoding of features from entire videos for the representation of
human actions has rarely been addressed. Instead, CNN work has focused on
approaches to fuse spatial and temporal networks, but these were typically
limited to processing shorter sequences. We present a new video representation,
called temporal linear encoding (TLE) and embedded inside of CNNs as a new
layer, which captures the appearance and motion throughout entire videos. It
encodes this aggregated information into a robust video feature representation,
via end-to-end learning. Advantages of TLEs are: (a) they encode the entire
video into a compact feature representation, learning the semantics and a
discriminative feature space; (b) they are applicable to all kinds of networks
like 2D and 3D CNNs for video classification; and (c) they model feature
interactions in a more expressive way and without loss of information. We
conduct experiments on two challenging human action datasets: HMDB51 and
UCF101. The experiments show that TLE outperforms current state-of-the-art
methods on both datasets.
