  The choice of a loss function is an important factor when training neural
networks for image restoration problems, such as single image super resolution.
The loss function should encourage natural and perceptually pleasing results. A
popular choice for a loss is a pre-trained network, such as VGG, which is used
as a feature extractor for computing the difference between restored and
reference images. However, such an approach has multiple drawbacks: it is
computationally expensive, requires regularization and hyper-parameter tuning,
and involves a large network trained on an unrelated task. Furthermore, it has
been observed that there is no single loss function that works best across all
applications and across different datasets. In this work, we instead propose to
train a set of loss functions that are application specific in nature. Our loss
function comprises a series of discriminators that are trained to detect and
penalize the presence of application-specific artifacts. We show that a single
natural image and corresponding distortions are sufficient to train our feature
extractor that outperforms state-of-the-art loss functions in applications like
single image super resolution, denoising, and JPEG artifact removal. Finally,
we conclude that an effective loss function does not have to be a good
predictor of perceived image quality, but instead needs to be specialized in
identifying the distortions for a given restoration method.
