  Given two consecutive frames, video interpolation aims at generating
intermediate frame(s) to form both spatially and temporally coherent video
sequences. While most existing methods focus on single-frame interpolation, we
propose an end-to-end convolutional neural network for variable-length
multi-frame video interpolation, where the motion interpretation and occlusion
reasoning are jointly modeled. We start by computing bi-directional optical
flow between the input images using a U-Net architecture. These flows are then
linearly combined at each time step to approximate the intermediate
bi-directional optical flows. These approximate flows, however, only work well
in locally smooth regions and produce artifacts around motion boundaries. To
address this shortcoming, we employ another U-Net to refine the approximated
flow and also predict soft visibility maps. Finally, the two input images are
warped and linearly fused to form each intermediate frame. By applying the
visibility maps to the warped images before fusion, we exclude the contribution
of occluded pixels to the interpolated intermediate frame to avoid artifacts.
Since none of our learned network parameters are time-dependent, our approach
is able to produce as many intermediate frames as needed. We use 1,132 video
clips with 240-fps, containing 300K individual video frames, to train our
network. Experimental results on several datasets, predicting different numbers
of interpolated frames, demonstrate that our approach performs consistently
better than existing methods.
