  Representations that can compactly and effectively capture the temporal
evolution of semantic content are important to computer vision and machine
learning algorithms that operate on multi-variate time-series data. We
investigate such representations motivated by the task of human action
recognition. Here each data instance is encoded by a multivariate feature (such
as via a deep CNN) where action dynamics are characterized by their variations
in time. As these features are often non-linear, we propose a novel pooling
method, kernelized rank pooling, that represents a given sequence compactly as
the pre-image of the parameters of a hyperplane in a reproducing kernel Hilbert
space, projections of data onto which captures their temporal order. We develop
this idea further and show that such a pooling scheme can be cast as an
order-constrained kernelized PCA objective. We then propose to use the
parameters of a kernelized low-rank feature subspace as the representation of
the sequences. We cast our formulation as an optimization problem on
generalized Grassmann manifolds and then solve it efficiently using Riemannian
optimization techniques. We present experiments on several action recognition
datasets using diverse feature modalities and demonstrate state-of-the-art
results.
