  In this work we propose a novel neural network architecture for the problem
of human action recognition in videos. The proposed architecture expresses the
processing steps of classical Fisher vector approaches, that is dimensionality
reduction by principal component analysis (PCA) projection, Gaussian mixture
model (GMM) and Fisher vector descriptor extraction, as network layers. By
contrast to other methods where these steps are performed consecutively and the
corresponding parameters are learned in an unsupervised manner, having them
defined as a single neural network allows us to refine the whole model
discriminatively in an end to end fashion. Furthermore, we show that the
proposed architecture can be used as a replacement for the fully connected
layers in popular convolutional networks achieving a comparable classification
performance, or even significantly surpassing the performance of similar
architectures while reducing the total number of trainable parameters by a
factor of 5. We show that our method achieves significant improvements in
comparison to the classical chain.
