  We propose a function-based temporal pooling method that captures the latent
structure of the video sequence data - e.g. how frame-level features evolve
over time in a video. We show how the parameters of a function that has been
fit to the video data can serve as a robust new video representation. As a
specific example, we learn a pooling function via ranking machines. By learning
to rank the frame-level features of a video in chronological order, we obtain a
new representation that captures the video-wide temporal dynamics of a video,
suitable for action recognition. Other than ranking functions, we explore
different parametric models that could also explain the temporal changes in
videos. The proposed functional pooling methods, and rank pooling in
particular, is easy to interpret and implement, fast to compute and effective
in recognizing a wide variety of actions. We evaluate our method on various
benchmarks for generic action, fine-grained action and gesture recognition.
Results show that rank pooling brings an absolute improvement of 7-10 average
pooling baseline. At the same time, rank pooling is compatible with and
complementary to several appearance and local motion based methods and
features, such as improved trajectories and deep learning features.
