  Single image super-resolution (SISR) algorithms reconstruct high-resolution
(HR) images with their low-resolution (LR) counterparts. It is desirable to
develop image quality assessment (IQA) methods that can not only evaluate and
compare SISR algorithms, but also guide their future development. In this
paper, we assess the quality of SISR generated images in a two-dimensional (2D)
space of structural fidelity versus statistical naturalness. This allows us to
observe the behaviors of different SISR algorithms as a tradeoff in the 2D
space. Specifically, SISR methods are traditionally designed to achieve high
structural fidelity but often sacrifice statistical naturalness, while recent
generative adversarial network (GAN) based algorithms tend to create more
natural-looking results but lose significantly on structural fidelity.
Furthermore, such a 2D evaluation can be easily fused to a scalar quality
prediction. Interestingly, we find that a simple linear combination of a
straightforward local structural fidelity and a global statistical naturalness
measures produce surprisingly accurate predictions of SISR image quality when
tested using public subject-rated SISR image datasets. Code of the proposed
SFSN model is publicly available at \url{https://github.com/weizhou-geek/SFSN}.
