  In this paper, we address a rain removal problem from a single image, even in
the presence of heavy rain and rain streak accumulation. Our core ideas lie in
the new rain image models and a novel deep learning architecture. We first
modify an existing model comprising a rain streak layer and a background layer,
by adding a binary map that locates rain streak regions. Second, we create a
new model consisting of a component representing rain streak accumulation
(where individual streaks cannot be seen, and thus visually similar to mist or
fog), and another component representing various shapes and directions of
overlapping rain streaks, which usually happen in heavy rain. Based on the
first model, we develop a multi-task deep learning architecture that learns the
binary rain streak map, the appearance of rain streaks, and the clean
background, which is our ultimate output. The additional binary map is
critically beneficial, since its loss function can provide additional strong
information to the network. To handle rain streak accumulation (again, a
phenomenon visually similar to mist or fog) and various shapes and directions
of overlapping rain streaks, we propose a recurrent rain detection and removal
network that removes rain streaks and clears up the rain accumulation
iteratively and progressively. In each recurrence of our method, a new
contextualized dilated network is developed to exploit regional contextual
information and outputs better representation for rain detection. The
evaluation on real images, particularly on heavy rain, shows the effectiveness
of our novel models and architecture, outperforming the state-of-the-art
methods significantly. Our codes and data sets will be publicly available.
