  Video object detection is a fundamental yet challenging task in computer
vision. One practical solution is to take advantage of temporal information
from the video and apply feature aggregation to enhance the object features in
each frame. Though effective, those existing methods always suffer from low
inference speeds because they use a fixed number of frames for feature
aggregation regardless of the input frame. Therefore, this paper aims to
improve the inference speed of the current feature aggregation-based video
object detectors while maintaining their performance. To achieve this goal, we
propose a vanilla dynamic aggregation module that adaptively selects the frames
for feature enhancement. Then, we extend the vanilla dynamic aggregation module
to a more effective and reconfigurable deformable version. Finally, we
introduce inplace distillation loss to improve the representations of objects
aggregated with fewer frames. Extensive experimental results validate the
effectiveness and efficiency of our proposed methods: On the ImageNet VID
benchmark, integrated with our proposed methods, FGFA and SELSA can improve the
inference speed by 31% and 76% respectively while getting comparable
performance on accuracy.
