  Recent years have witnessed tremendous progress in single image
super-resolution (SISR) owing to the deployment of deep convolutional neural
networks (CNNs). For most existing methods, the computational cost of each SISR
model is irrelevant to local image content, hardware platform and application
scenario. Nonetheless, content and resource adaptive model is more preferred,
and it is encouraging to apply simpler and efficient networks to the easier
regions with less details and the scenarios with restricted efficiency
constraints. In this paper, we take a step forward to address this issue by
leveraging the adaptive inference networks for deep SISR (AdaDSR). In
particular, our AdaDSR involves an SISR model as backbone and a lightweight
adapter module which takes image features and resource constraint as input and
predicts a map of local network depth. Adaptive inference can then be performed
with the support of efficient sparse convolution, where only a fraction of the
layers in the backbone is performed at a given position according to its
predicted depth. The network learning can be formulated as the joint
optimization of reconstruction and network depth losses. In the inference
stage, the average depth can be flexibly tuned to meet a range of efficiency
constraints. Experiments demonstrate the effectiveness and adaptability of our
AdaDSR in contrast to its counterparts (e.g., EDSR and RCAN).
