  In this paper, we propose an end-to-end 3D CNN for action detection and
segmentation in videos. The proposed architecture is a unified deep network
that is able to recognize and localize action based on 3D convolution features.
A video is first divided into equal length clips and next for each clip a set
of tube proposals are generated based on 3D CNN features. Finally, the tube
proposals of different clips are linked together and spatio-temporal action
detection is performed using these linked video proposals. This top-down action
detection approach explicitly relies on a set of good tube proposals to perform
well and training the bounding box regression usually requires a large number
of annotated samples. To remedy this, we further extend the 3D CNN to an
encoder-decoder structure and formulate the localization problem as action
segmentation. The foreground regions (i.e. action regions) for each frame are
segmented first then the segmented foreground maps are used to generate the
bounding boxes. This bottom-up approach effectively avoids tube proposal
generation by leveraging the pixel-wise annotations of segmentation. The
segmentation framework also can be readily applied to a general problem of
video object segmentation. Extensive experiments on several video datasets
demonstrate the superior performance of our approach for action detection and
video object segmentation compared to the state-of-the-arts.
