  High perceptual quality and low distortion degree are two important goals in
image restoration tasks such as super-resolution (SR). Most of the existing SR
methods aim to achieve these goals by minimizing the corresponding yet
conflicting losses, such as the $\ell_1$ loss and the adversarial loss.
Unfortunately, the commonly used gradient-based optimizers, such as Adam, are
hard to balance these objectives due to the opposite gradient decent directions
of the contradictory losses. In this paper, we formulate the
perception-distortion trade-off in SR as a multi-objective optimization problem
and develop a new optimizer by integrating the gradient-free evolutionary
algorithm (EA) with gradient-based Adam, where EA and Adam focus on the
divergence and convergence of the optimization directions respectively. As a
result, a population of optimal models with different perception-distortion
preferences is obtained. We then design a fusion network to merge these models
into a single stronger one for an effective perception-distortion trade-off.
Experiments demonstrate that with the same backbone network, the
perception-distortion balanced SR model trained by our method can achieve
better perceptual quality than its competitors while attaining better
reconstruction fidelity. Codes and models can be found at
https://github.com/csslc/EA-Adam.
