  Transferring existing image-based detectors to the video is non-trivial since
the quality of frames is always deteriorated by part occlusion, rare pose, and
motion blur. Previous approaches exploit to propagate and aggregate features
across video frames by using optical flow-warping. However, directly applying
image-level optical flow onto the high-level features might not establish
accurate spatial correspondences. Therefore, a novel module called Learnable
Spatio-Temporal Sampling (LSTS) has been proposed to learn semantic-level
correspondences among adjacent frame features accurately. The sampled locations
are first randomly initialized, then updated iteratively to find better spatial
correspondences guided by detection supervision progressively. Besides,
Sparsely Recursive Feature Updating (SRFU) module and Dense Feature Aggregation
(DFA) module are also introduced to model temporal relations and enhance
per-frame features, respectively. Without bells and whistles, the proposed
method achieves state-of-the-art performance on the ImageNet VID dataset with
less computational complexity and real-time speed. Code will be made available
at https://github.com/jiangzhengkai/LSTS.
