  Recent success in the field of single image super-resolution (SISR) is
achieved by optimizing deep convolutional neural networks (CNNs) in the image
space with the L1 or L2 loss. However, when trained with these loss functions,
models usually fail to recover sharp edges present in the high-resolution (HR)
images for the reason that the model tends to give a statistical average of
potential HR solutions. During our research, we observe that gradient maps of
images generated by the models trained with the L1 or L2 loss have
significantly lower variance than the gradient maps of the original
high-resolution images. In this work, we propose to alleviate the above issue
by introducing a structure-enhancing loss function, coined Gradient Variance
(GV) loss, and generate textures with perceptual-pleasant details.
Specifically, during the training of the model, we extract patches from the
gradient maps of the target and generated output, calculate the variance of
each patch and form variance maps for these two images. Further, we minimize
the distance between the computed variance maps to enforce the model to produce
high variance gradient maps that will lead to the generation of high-resolution
images with sharper edges. Experimental results show that the GV loss can
significantly improve both Structure Similarity (SSIM) and peak signal-to-noise
ratio (PSNR) performance of existing image super-resolution (SR) deep learning
models.
