  Images acquired from rainy scenes usually suffer from bad visibility which
may damage the performance of computer vision applications. The rainy scenarios
can be categorized into two classes: moderate rain and heavy rain scenes.
Moderate rain scene mainly consists of rain streaks while heavy rain scene
contains both rain streaks and the veiling effect (similar to haze). Although
existing methods have achieved excellent performance on these two cases
individually, it still lacks a general architecture to address both heavy rain
and moderate rain scenarios effectively. In this paper, we construct a
hierarchical multi-direction representation network by using the contourlet
transform (CT) to address both moderate rain and heavy rain scenarios. The CT
divides the image into the multi-direction subbands (MS) and the semantic
subband (SS). First, the rain streak information is retrieved to the MS based
on the multi-orientation property of the CT. Second, a hierarchical
architecture is proposed to reconstruct the background information including
damaged semantic information and the veiling effect in the SS. Last, the
multi-level subband discriminator with the feedback error map is proposed. By
this module, all subbands can be well optimized. This is the first architecture
that can address both of the two scenarios effectively. The code is available
in https://github.com/cctakaet/ContourletNet-BMVC2021.
