  Video frame synthesis, which consists of interpolation and extrapolation, is
an essential video processing technique that can be applied to various
scenarios. However, most existing methods cannot handle small objects or large
motion well, especially in high-resolution videos such as 4K videos. To
eliminate such limitations, we introduce a neighbor correspondence matching
(NCM) algorithm for flow-based frame synthesis. Since the current frame is not
available in video frame synthesis, NCM is performed in a
current-frame-agnostic fashion to establish multi-scale correspondences in the
spatial-temporal neighborhoods of each pixel. Based on the powerful motion
representation capability of NCM, we further propose to estimate intermediate
flows for frame synthesis in a heterogeneous coarse-to-fine scheme.
Specifically, the coarse-scale module is designed to leverage neighbor
correspondences to capture large motion, while the fine-scale module is more
computationally efficient to speed up the estimation process. Both modules are
trained progressively to eliminate the resolution gap between training dataset
and real-world videos. Experimental results show that NCM achieves
state-of-the-art performance on several benchmarks. In addition, NCM can be
applied to various practical scenarios such as video compression to achieve
better performance.
