  Image super-resolution (SR) has witnessed extensive neural network designs
from CNN to transformer architectures. However, prevailing SR models suffer
from prohibitive memory footprint and intensive computations, which limits
further deployment on edge devices. This work investigates the potential of
network pruning for super-resolution to take advantage of off-the-shelf network
designs and reduce the underlying computational overhead. Two main challenges
remain in applying pruning methods for SR. First, the widely-used filter
pruning technique reflects limited granularity and restricted adaptability to
diverse network structures. Second, existing pruning methods generally operate
upon a pre-trained network for the sparse structure determination, hard to get
rid of dense model training in the traditional SR paradigm. To address these
challenges, we adopt unstructured pruning with sparse models directly trained
from scratch. Specifically, we propose a novel Iterative Soft
Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a
randomly initialized network at each iteration and tweaking unimportant weights
with a small amount proportional to the magnitude scale on-the-fly. We observe
that the proposed ISS-P can dynamically learn sparse structures adapting to the
optimization process and preserve the sparse model's trainability by yielding a
more regularized gradient throughput. Experiments on benchmark datasets
demonstrate the effectiveness of the proposed ISS-P over diverse network
architectures. Code is available at
https://github.com/Jiamian-Wang/Iterative-Soft-Shrinkage-SR
