  In this work, we present novel temporal encoding methods for action and
activity classification by extending the unsupervised rank pooling temporal
encoding method in two ways. First, we present "discriminative rank pooling" in
which the shared weights of our video representation and the parameters of the
action classifiers are estimated jointly for a given training dataset of
labelled vector sequences using a bilevel optimization formulation of the
learning problem. When the frame level features vectors are obtained from a
convolutional neural network (CNN), we rank pool the network activations and
jointly estimate all parameters of the model, including CNN filters and
fully-connected weights, in an end-to-end manner which we coined as "end-to-end
trainable rank pooled CNN". Importantly, this model can make use of any
existing convolutional neural network architecture (e.g., AlexNet or VGG)
without modification or introduction of additional parameters. Then, we extend
rank pooling to a high capacity video representation, called "hierarchical rank
pooling". Hierarchical rank pooling consists of a network of rank pooling
functions, which encode temporal semantics over arbitrary long video clips
based on rich frame level features. By stacking non-linear feature functions
and temporal sub-sequence encoders one on top of the other, we build a high
capacity encoding network of the dynamic behaviour of the video. The resulting
video representation is a fixed-length feature vector describing the entire
video clip that can be used as input to standard machine learning classifiers.
We demonstrate our approach on the task of action and activity recognition.
Obtained results are comparable to state-of-the-art methods on three important
activity recognition benchmarks with classification performance of 76.7% mAP on
Hollywood2, 69.4% on HMDB51, and 93.6% on UCF101.
