  We propose a hierarchical approach to multi-action recognition that performs
joint classification and segmentation. A given video (containing several
consecutive actions) is processed via a sequence of overlapping temporal
windows. Each frame in a temporal window is represented through selective
low-level spatio-temporal features which efficiently capture relevant local
dynamics. Features from each window are represented as a Fisher vector, which
captures first and second order statistics. Instead of directly classifying
each Fisher vector, it is converted into a vector of class probabilities. The
final classification decision for each frame is then obtained by integrating
the class probabilities at the frame level, which exploits the overlapping of
the temporal windows. Experiments were performed on two datasets: s-KTH (a
stitched version of the KTH dataset to simulate multi-actions), and the
challenging CMU-MMAC dataset. On s-KTH, the proposed approach achieves an
accuracy of 85.0%, significantly outperforming two recent approaches based on
GMMs and HMMs which obtained 78.3% and 71.2%, respectively. On CMU-MMAC, the
proposed approach achieves an accuracy of 40.9%, outperforming the GMM and HMM
approaches which obtained 33.7% and 38.4%, respectively. Furthermore, the
proposed system is on average 40 times faster than the GMM based approach.
