  Feature pyramids and iterative refinement have recently led to great progress
in optical flow estimation. However, downsampling in feature pyramids can cause
blending of foreground objects with the background, which will mislead
subsequent decisions in the iterative processing. The results are missing
details especially in the flow of thin and of small structures. We propose a
novel Residual Feature Pyramid Module (RFPM) which retains important details in
the feature map without changing the overall iterative refinement design of the
optical flow estimation. RFPM incorporates a residual structure between
multiple feature pyramids into a downsampling module that corrects the blending
of objects across boundaries. We demonstrate how to integrate our module with
two state-of-the-art iterative refinement architectures. Results show that our
RFPM visibly reduces flow errors and improves state-of-art performance in the
clean pass of Sintel, and is one of the top-performing methods in KITTI.
According to the particular modular structure of RFPM, we introduce a special
transfer learning approach that can dramatically decrease the training time
compared to a typical full optical flow training schedule on multiple datasets.
