  With the recent advancement in deep learning, we have witnessed a great
progress in single image super-resolution. However, due to the significant
information loss of the image downscaling process, it has become extremely
challenging to further advance the state-of-the-art, especially for large
upscaling factors. This paper explores a new research direction in super
resolution, called reference-conditioned super-resolution, in which a reference
image containing desired high-resolution texture details is provided besides
the low-resolution image. We focus on transferring the high-resolution texture
from reference images to the super-resolution process without the constraint of
content similarity between reference and target images, which is a key
difference from previous example-based methods. Inspired by recent work on
image stylization, we address the problem via neural texture transfer. We
design an end-to-end trainable deep model which generates detail enriched
results by adaptively fusing the content from the low-resolution image with the
texture patterns from the reference image. We create a benchmark dataset for
the general research of reference-based super-resolution, which contains
reference images paired with low-resolution inputs with varying degrees of
similarity. Both objective and subjective evaluations demonstrate the great
potential of using reference images as well as the superiority of our results
over other state-of-the-art methods.
