  Rain removal aims to remove the rain streaks on rain images. The
state-of-the-art methods are mostly based on Convolutional Neural
Network~(CNN). However, as CNN is not equivariant to object rotation, these
methods are unsuitable for dealing with the tilted rain streaks. To tackle this
problem, we propose Deep Symmetry Enhanced Network~(DSEN) that is able to
explicitly extract the rotation equivariant features from rain images. In
addition, we design a self-refining mechanism to remove the accumulated rain
streaks in a coarse-to-fine manner. This mechanism reuses DSEN with a novel
information link which passes the gradient flow to the higher stages. Extensive
experiments on both synthetic and real-world rain images show that our
self-refining DSEN yields the top performance.
