  In Video Object Detection (VID), a common practice is to leverage the rich
temporal contexts from the video to enhance the object representations in each
frame. Existing methods treat the temporal contexts obtained from different
objects indiscriminately and ignore their different identities. While
intuitively, aggregating local views of the same object in different frames may
facilitate a better understanding of the object. Thus, in this paper, we aim to
enable the model to focus on the identity-consistent temporal contexts of each
object to obtain more comprehensive object representations and handle the rapid
object appearance variations such as occlusion, motion blur, etc. However,
realizing this goal on top of existing VID models faces low-efficiency problems
due to their redundant region proposals and nonparallel frame-wise prediction
manner. To aid this, we propose ClipVID, a VID model equipped with
Identity-Consistent Aggregation (ICA) layers specifically designed for mining
fine-grained and identity-consistent temporal contexts. It effectively reduces
the redundancies through the set prediction strategy, making the ICA layers
very efficient and further allowing us to design an architecture that makes
parallel clip-wise predictions for the whole video clip. Extensive experimental
results demonstrate the superiority of our method: a state-of-the-art (SOTA)
performance (84.7% mAP) on the ImageNet VID dataset while running at a speed
about 7x faster (39.3 fps) than previous SOTAs.
