  Detecting representative frames in videos based on human actions is quite
challenging because of the combined factors of human pose in action and the
background. This paper addresses this problem and formulates the key frame
detection as one of finding the video frames that optimally maximally
contribute to differentiating the underlying action category from all other
categories. To this end, we introduce a deep two-stream ConvNet for key frame
detection in videos that learns to directly predict the location of key frames.
Our key idea is to automatically generate labeled data for the CNN learning
using a supervised linear discriminant method. While the training data is
generated taking many different human action videos into account, the trained
CNN can predict the importance of frames from a single video. We specify a new
ConvNet framework, consisting of a summarizer and discriminator. The summarizer
is a two-stream ConvNet aimed at, first, capturing the appearance and motion
features of video frames, and then encoding the obtained appearance and motion
features for video representation. The discriminator is a fitting function
aimed at distinguishing between the key frames and others in the video. We
conduct experiments on a challenging human action dataset UCF101 and show that
our method can detect key frames with high accuracy.
