  Video restoration aims at restoring multiple high-quality frames from
multiple low-quality frames. Existing video restoration methods generally fall
into two extreme cases, i.e., they either restore all frames in parallel or
restore the video frame by frame in a recurrent way, which would result in
different merits and drawbacks. Typically, the former has the advantage of
temporal information fusion. However, it suffers from large model size and
intensive memory consumption; the latter has a relatively small model size as
it shares parameters across frames; however, it lacks long-range dependency
modeling ability and parallelizability. In this paper, we attempt to integrate
the advantages of the two cases by proposing a recurrent video restoration
transformer, namely RVRT. RVRT processes local neighboring frames in parallel
within a globally recurrent framework which can achieve a good trade-off
between model size, effectiveness, and efficiency. Specifically, RVRT divides
the video into multiple clips and uses the previously inferred clip feature to
estimate the subsequent clip feature. Within each clip, different frame
features are jointly updated with implicit feature aggregation. Across
different clips, the guided deformable attention is designed for clip-to-clip
alignment, which predicts multiple relevant locations from the whole inferred
clip and aggregates their features by the attention mechanism. Extensive
experiments on video super-resolution, deblurring, and denoising show that the
proposed RVRT achieves state-of-the-art performance on benchmark datasets with
balanced model size, testing memory and runtime.
