  Most of the current action recognition algorithms are based on deep networks
which stack multiple convolutional, pooling and fully connected layers. While
convolutional and fully connected operations have been widely studied in the
literature, the design of pooling operations that handle action recognition,
with different sources of temporal granularity in action categories, has
comparatively received less attention, and existing solutions rely mainly on
max or averaging operations. The latter are clearly powerless to fully exhibit
the actual temporal granularity of action categories and thereby constitute a
bottleneck in classification performances. In this paper, we introduce a novel
hierarchical pooling design that captures different levels of temporal
granularity in action recognition. Our design principle is coarse-to-fine and
achieved using a tree-structured network; as we traverse this network top-down,
pooling operations are getting less invariant but timely more resolute and well
localized. Learning the combination of operations in this network -- which best
fits a given ground-truth -- is obtained by solving a constrained minimization
problem whose solution corresponds to the distribution of weights that capture
the contribution of each level (and thereby temporal granularity) in the global
hierarchical pooling process. Besides being principled and well grounded, the
proposed hierarchical pooling is also video-length and resolution agnostic.
Extensive experiments conducted on the challenging UCF-101, HMDB-51 and
JHMDB-21 databases corroborate all these statements.
