  Obtaining the ground truth labels from a video is challenging since the
manual annotation of pixel-wise flow labels is prohibitively expensive and
laborious. Besides, existing approaches try to adapt the trained model on
synthetic datasets to authentic videos, which inevitably suffers from domain
discrepancy and hinders the performance for real-world applications. To solve
these problems, we propose RealFlow, an Expectation-Maximization based
framework that can create large-scale optical flow datasets directly from any
unlabeled realistic videos. Specifically, we first estimate optical flow
between a pair of video frames, and then synthesize a new image from this pair
based on the predicted flow. Thus the new image pairs and their corresponding
flows can be regarded as a new training set. Besides, we design a Realistic
Image Pair Rendering (RIPR) module that adopts softmax splatting and
bi-directional hole filling techniques to alleviate the artifacts of the image
synthesis. In the E-step, RIPR renders new images to create a large quantity of
training data. In the M-step, we utilize the generated training data to train
an optical flow network, which can be used to estimate optical flows in the
next E-step. During the iterative learning steps, the capability of the flow
network is gradually improved, so is the accuracy of the flow, as well as the
quality of the synthesized dataset. Experimental results show that RealFlow
outperforms previous dataset generation methods by a considerably large margin.
Moreover, based on the generated dataset, our approach achieves
state-of-the-art performance on two standard benchmarks compared with both
supervised and unsupervised optical flow methods. Our code and dataset are
available at https://github.com/megvii-research/RealFlow
