  Single image super-resolution (SISR) is an ill-posed problem with an
indeterminate number of valid solutions. Solving this problem with neural
networks would require access to extensive experience, either presented as a
large training set over natural images or a condensed representation from
another pre-trained network. Perceptual loss functions, which belong to the
latter category, have achieved breakthrough success in SISR and several other
computer vision tasks. While perceptual loss plays a central role in the
generation of photo-realistic images, it also produces undesired pattern
artifacts in the super-resolved outputs. In this paper, we show that the root
cause of these pattern artifacts can be traced back to a mismatch between the
pre-training objective of perceptual loss and the super-resolution objective.
To address this issue, we propose to augment the existing perceptual loss
formulation with a novel content loss function that uses the latent features of
a discriminator network to filter the unwanted artifacts across several levels
of adversarial similarity. Further, our modification has a stabilizing effect
on non-convex optimization in adversarial training. The proposed approach
offers notable gains in perceptual quality based on an extensive human
evaluation study and a competent reconstruction fidelity when tested on
objective evaluation metrics.
