  Two-stream networks have been very successful for solving the problem of
action detection. However, prior work using two-stream networks train both
streams separately, which prevents the network from exploiting regularities
between the two streams. Moreover, unlike the visual stream, the dominant forms
of optical flow computation typically do not maximally exploit GPU parallelism.
We present a real-time end-to-end trainable two-stream network for action
detection. First, we integrate the optical flow computation in our framework by
using Flownet2. Second, we apply early fusion for the two streams and train the
whole pipeline jointly end-to-end. Finally, for better network initialization,
we transfer from the task of action recognition to action detection by
pre-training our framework using the recently released large-scale Kinetics
dataset. Our experimental results show that training the pipeline jointly
end-to-end with fine-tuning the optical flow for the objective of action
detection improves detection performance significantly. Additionally, we
observe an improvement when initializing with parameters pre-trained using
Kinetics. Last, we show that by integrating the optical flow computation, our
framework is more efficient, running at real-time speeds (up to 31 fps).
