  Multi-scale architectures have shown effectiveness in a variety of tasks
thanks to appealing cross-scale complementarity. However, existing
architectures treat different scale features equally without considering the
scale-specific characteristics, \textit{i.e.}, the within-scale characteristics
are ignored in the architecture design. In this paper, we reveal this missing
piece for multi-scale architecture design and accordingly propose a novel
Multi-Scale Adaptive Network (MSANet) for single image denoising. Specifically,
MSANet simultaneously embraces the within-scale characteristics and the
cross-scale complementarity thanks to three novel neural blocks, \textit{i.e.},
adaptive feature block (AFeB), adaptive multi-scale block (AMB), and adaptive
fusion block (AFuB). In brief, AFeB is designed to adaptively preserve image
details and filter noises, which is highly expected for the features with mixed
details and noises. AMB could enlarge the receptive field and aggregate the
multi-scale information, which meets the need of contextually informative
features. AFuB devotes to adaptively sampling and transferring the features
from one scale to another scale, which fuses the multi-scale features with
varying characteristics from coarse to fine. Extensive experiments on both
three real and six synthetic noisy image datasets show the superiority of
MSANet compared with 12 methods. The code could be accessed from
https://github.com/XLearning-SCU/2022-NeurIPS-MSANet.
