  Image and video classification research has made great progress through the
development of handcrafted local features and learning based features. These
two architectures were proposed roughly at the same time and have flourished at
overlapping stages of history. However, they are typically viewed as distinct
approaches. In this paper, we emphasize their structural similarities and show
how such a unified view helps us in designing features that balance efficiency
and effectiveness. As an example, we study the problem of designing efficient
video feature learning algorithms for action recognition.
  We approach this problem by first showing that local handcrafted features and
Convolutional Neural Networks (CNNs) share the same convolution-pooling network
structure. We then propose a two-stream Convolutional ISA (ConvISA) that adopts
the convolution-pooling structure of the state-of-the-art handcrafted video
feature with greater modeling capacities and a cost-effective training
algorithm. Through custom designed network structures for pixels and optical
flow, our method also reflects distinctive characteristics of these two data
sources.
  Our experimental results on standard action recognition benchmarks show that
by focusing on the structure of CNNs, rather than end-to-end training methods,
we are able to design an efficient and powerful video feature learning
algorithm.
