  Deep video models, for example, 3D CNNs or video transformers, have achieved
promising performance on sparse video tasks, i.e., predicting one result per
video. However, challenges arise when adapting existing deep video models to
dense video tasks, i.e., predicting one result per frame. Specifically, these
models are expensive for deployment, less effective when handling redundant
frames, and difficult to capture long-range temporal correlations. To overcome
these issues, we propose a Temporal Dilated Video Transformer (TDViT) that
consists of carefully designed temporal dilated transformer blocks (TDTB). TDTB
can efficiently extract spatiotemporal representations and effectively
alleviate the negative effect of temporal redundancy. Furthermore, by using
hierarchical TDTBs, our approach obtains an exponentially expanded temporal
receptive field and therefore can model long-range dynamics. Extensive
experiments are conducted on two different dense video benchmarks, i.e.,
ImageNet VID for video object detection and YouTube VIS for video instance
segmentation. Excellent experimental results demonstrate the superior
efficiency, effectiveness, and compatibility of our method. The code is
available at https://github.com/guanxiongsun/vfe.pytorch.
