  Optical flow estimation in the rainy scenes is challenging due to background
degradation introduced by rain streaks and rain accumulation effects in the
scene. Rain accumulation effect refers to poor visibility of remote objects due
to the intense rainfall. Most existing optical flow methods are erroneous when
applied to rain sequences because the conventional brightness constancy
constraint (BCC) and gradient constancy constraint (GCC) generally break down
in this situation. Based on the observation that the RGB color channels receive
raindrop radiance equally, we introduce a residue channel as a new data
constraint to reduce the effect of rain streaks. To handle rain accumulation,
our method decomposes the image into a piecewise-smooth background layer and a
high-frequency detail layer. It also enforces the BCC on the background layer
only. Results on both synthetic dataset and real images show that our algorithm
outperforms existing methods on different types of rain sequences. To our
knowledge, this is the first optical flow method specifically dealing with
rain.
