  In this paper, a novel encoding scheme combining Fisher vector and
bag-of-words encodings has been proposed for recognizing action in videos. The
proposed Hyper-Fisher vector encoding is sum of local Fisher vectors which are
computed based on the traditional Bag-of-Words (BoW) encoding. Thus, the
proposed encoding is simple and yet an effective representation over the
traditional Fisher Vector encoding. By extensive evaluation on challenging
action recognition datasets, viz., Youtube, Olympic Sports, UCF50 and HMDB51,
we show that the proposed Hyper-Fisher Vector encoding improves the recognition
performance by around 2-3% compared to the improved Fisher Vector encoding. We
also perform experiments to show that the performance of the Hyper-Fisher
Vector is robust to the dictionary size of the BoW encoding.
