  Recently, very deep convolutional neural networks (CNNs) have been attracting
considerable attention in image restoration. However, as the depth grows, the
long-term dependency problem is rarely realized for these very deep models,
which results in the prior states/layers having little influence on the
subsequent ones. Motivated by the fact that human thoughts have persistency, we
propose a very deep persistent memory network (MemNet) that introduces a memory
block, consisting of a recursive unit and a gate unit, to explicitly mine
persistent memory through an adaptive learning process. The recursive unit
learns multi-level representations of the current state under different
receptive fields. The representations and the outputs from the previous memory
blocks are concatenated and sent to the gate unit, which adaptively controls
how much of the previous states should be reserved, and decides how much of the
current state should be stored. We apply MemNet to three image restoration
tasks, i.e., image denosing, super-resolution and JPEG deblocking.
Comprehensive experiments demonstrate the necessity of the MemNet and its
unanimous superiority on all three tasks over the state of the arts. Code is
available at https://github.com/tyshiwo/MemNet.
