  Recent works have shown that the computational efficiency of video
recognition can be significantly improved by reducing the spatial redundancy.
As a representative work, the adaptive focus method (AdaFocus) has achieved a
favorable trade-off between accuracy and inference speed by dynamically
identifying and attending to the informative regions in each video frame.
However, AdaFocus requires a complicated three-stage training pipeline
(involving reinforcement learning), leading to slow convergence and is
unfriendly to practitioners. This work reformulates the training of AdaFocus as
a simple one-stage algorithm by introducing a differentiable
interpolation-based patch selection operation, enabling efficient end-to-end
optimization. We further present an improved training scheme to address the
issues introduced by the one-stage formulation, including the lack of
supervision, input diversity and training stability. Moreover, a
conditional-exit technique is proposed to perform temporal adaptive computation
on top of AdaFocus without additional training. Extensive experiments on six
benchmark datasets (i.e., ActivityNet, FCVID, Mini-Kinetics,
Something-Something V1&V2, and Jester) demonstrate that our model significantly
outperforms the original AdaFocus and other competitive baselines, while being
considerably more simple and efficient to train. Code is available at
https://github.com/LeapLabTHU/AdaFocusV2.
