  Current CNN-based super-resolution (SR) methods process all locations equally
with computational resources being uniformly assigned in space. However, since
missing details in low-resolution (LR) images mainly exist in regions of edges
and textures, less computational resources are required for those flat regions.
Therefore, existing CNN-based methods involve redundant computation in flat
regions, which increases their computational cost and limits their applications
on mobile devices. In this paper, we explore the sparsity in image SR to
improve inference efficiency of SR networks. Specifically, we develop a Sparse
Mask SR (SMSR) network to learn sparse masks to prune redundant computation.
Within our SMSR, spatial masks learn to identify "important" regions while
channel masks learn to mark redundant channels in those "unimportant" regions.
Consequently, redundant computation can be accurately localized and skipped
while maintaining comparable performance. It is demonstrated that our SMSR
achieves state-of-the-art performance with 41%/33%/27% FLOPs being reduced for
x2/3/4 SR. Code is available at: https://github.com/LongguangWang/SMSR.
