  Image restoration is the task of aiming to obtain a high-quality image from a
corrupt input image, such as deblurring and deraining. In image restoration, it
is typically necessary to maintain a complex balance between spatial details
and contextual information. Although a multi-stage network can optimally
balance these competing goals and achieve significant performance, this also
increases the system's complexity. In this paper, we propose a mountain-shaped
single-stage design base on a simple U-Net architecture, which removes or
replaces unnecessary nonlinear activation functions to achieve the above
balance with low system complexity. Specifically, we propose a feature fusion
middleware (FFM) mechanism as an information exchange component between the
encoder-decoder architectural levels. It seamlessly integrates upper-layer
information into the adjacent lower layer, sequentially down to the lowest
layer. Finally, all information is fused into the original image resolution
manipulation level. This preserves spatial details and integrates contextual
information, ensuring high-quality image restoration. In addition, we propose a
multi-head attention middle block (MHAMB) as a bridge between the encoder and
decoder to capture more global information and surpass the limitations of the
receptive field of CNNs. Extensive experiments demonstrate that our approach,
named as M3SNet, outperforms previous state-of-the-art models while using less
than half the computational costs, for several image restoration tasks, such as
image deraining and deblurring.
