  Deep neural networks have achieved remarkable success in single image
super-resolution (SISR). The computing and memory requirements of these methods
have hindered their application to broad classes of real devices with limited
computing power, however. One approach to this problem has been lightweight
network architectures that bal- ance the super-resolution performance and the
computation burden. In this study, we revisit this problem from an orthog- onal
view, and propose a novel learning strategy to maxi- mize the pixel-wise
fitting capacity of a given lightweight network architecture. Considering that
the initial capacity of the lightweight network is very limited, we present an
adaptive importance learning scheme for SISR that trains the network with an
easy-to-complex paradigm by dynam- ically updating the importance of image
pixels on the basis of the training loss. Specifically, we formulate the
network training and the importance learning into a joint optimization problem.
With a carefully designed importance penalty function, the importance of
individual pixels can be gradu- ally increased through solving a convex
optimization problem. The training process thus begins with pixels that are
easy to reconstruct, and gradually proceeds to more complex pixels as fitting
improves.
