  Downsampling is one of the most basic image processing operations. Improper
spatio-temporal downsampling applied on videos can cause aliasing issues such
as moir\'e patterns in space and the wagon-wheel effect in time. Consequently,
the inverse task of upscaling a low-resolution, low frame-rate video in space
and time becomes a challenging ill-posed problem due to information loss and
aliasing artifacts. In this paper, we aim to solve the space-time aliasing
problem by learning a spatio-temporal downsampler. Towards this goal, we
propose a neural network framework that jointly learns spatio-temporal
downsampling and upsampling. It enables the downsampler to retain the key
patterns of the original video and maximizes the reconstruction performance of
the upsampler. To make the downsamping results compatible with popular image
and video storage formats, the downsampling results are encoded to uint8 with a
differentiable quantization layer. To fully utilize the space-time
correspondences, we propose two novel modules for explicit temporal propagation
and space-time feature rearrangement. Experimental results show that our
proposed method significantly boosts the space-time reconstruction quality by
preserving spatial textures and motion patterns in both downsampling and
upscaling. Moreover, our framework enables a variety of applications, including
arbitrary video resampling, blurry frame reconstruction, and efficient video
storage.
