  Video restoration (e.g., video super-resolution) aims to restore high-quality
frames from low-quality frames. Different from single image restoration, video
restoration generally requires to utilize temporal information from multiple
adjacent but usually misaligned video frames. Existing deep methods generally
tackle with this by exploiting a sliding window strategy or a recurrent
architecture, which either is restricted by frame-by-frame restoration or lacks
long-range modelling ability. In this paper, we propose a Video Restoration
Transformer (VRT) with parallel frame prediction and long-range temporal
dependency modelling abilities. More specifically, VRT is composed of multiple
scales, each of which consists of two kinds of modules: temporal mutual self
attention (TMSA) and parallel warping. TMSA divides the video into small clips,
on which mutual attention is applied for joint motion estimation, feature
alignment and feature fusion, while self attention is used for feature
extraction. To enable cross-clip interactions, the video sequence is shifted
for every other layer. Besides, parallel warping is used to further fuse
information from neighboring frames by parallel feature warping. Experimental
results on five tasks, including video super-resolution, video deblurring,
video denoising, video frame interpolation and space-time video
super-resolution, demonstrate that VRT outperforms the state-of-the-art methods
by large margins ($\textbf{up to 2.16dB}$) on fourteen benchmark datasets.
