  We address the problem of activity detection in continuous, untrimmed video
streams. This is a difficult task that requires extracting meaningful
spatio-temporal features to capture activities, accurately localizing the start
and end times of each activity. We introduce a new model, Region Convolutional
3D Network (R-C3D), which encodes the video streams using a three-dimensional
fully convolutional network, then generates candidate temporal regions
containing activities, and finally classifies selected regions into specific
activities. Computation is saved due to the sharing of convolutional features
between the proposal and the classification pipelines. The entire model is
trained end-to-end with jointly optimized localization and classification
losses. R-C3D is faster than existing methods (569 frames per second on a
single Titan X Maxwell GPU) and achieves state-of-the-art results on THUMOS'14.
We further demonstrate that our model is a general activity detection framework
that does not rely on assumptions about particular dataset properties by
evaluating our approach on ActivityNet and Charades. Our code is available at
http://ai.bu.edu/r-c3d/.
