  Deep learning-based algorithms have greatly improved the performance of
remote sensing image (RSI) super-resolution (SR). However, increasing network
depth and parameters cause a huge burden of computing and storage. Directly
reducing the depth or width of existing models results in a large performance
drop. We observe that the SR difficulty of different regions in an RSI varies
greatly, and existing methods use the same deep network to process all regions
in an image, resulting in a waste of computing resources. In addition, existing
SR methods generally predefine integer scale factors and cannot perform
stepless SR, i.e., a single model can deal with any potential scale factor.
Retraining the model on each scale factor wastes considerable computing
resources and model storage space. To address the above problems, we propose a
saliency-aware dynamic routing network (SalDRN) for lightweight and stepless SR
of RSIs. First, we introduce visual saliency as an indicator of region-level SR
difficulty and integrate a lightweight saliency detector into the SalDRN to
capture pixel-level visual characteristics. Then, we devise a saliency-aware
dynamic routing strategy that employs path selection switches to adaptively
select feature extraction paths of appropriate depth according to the SR
difficulty of sub-image patches. Finally, we propose a novel lightweight
stepless upsampling module whose core is an implicit feature function for
realizing mapping from low-resolution feature space to high-resolution feature
space. Comprehensive experiments verify that the SalDRN can achieve a good
trade-off between performance and complexity. The code is available at
\url{https://github.com/hanlinwu/SalDRN}.
