  We present a deep-learning framework for real-time multiple spatio-temporal
(S/T) action localisation, classification and early prediction. Current
state-of-the-art approaches work offline and are too slow to be useful in real-
world settings. To overcome their limitations we introduce two major
developments. Firstly, we adopt real-time SSD (Single Shot MultiBox Detector)
convolutional neural networks to regress and classify detection boxes in each
video frame potentially containing an action of interest. Secondly, we design
an original and efficient online algorithm to incrementally construct and label
`action tubes' from the SSD frame level detections. As a result, our system is
not only capable of performing S/T detection in real time, but can also perform
early action prediction in an online fashion. We achieve new state-of-the-art
results in both S/T action localisation and early action prediction on the
challenging UCF101-24 and J-HMDB-21 benchmarks, even when compared to the top
offline competitors. To the best of our knowledge, ours is the first real-time
(up to 40fps) system able to perform online S/T action localisation and early
action prediction on the untrimmed videos of UCF101-24.
