  Deep models have achieved significant process on single image
super-resolution (SISR) tasks, in particular large models with large kernel
($3\times3$ or more). However, the heavy computational footprint of such models
prevents their deployment in real-time, resource-constrained environments.
Conversely, $1\times1$ convolutions bring substantial computational efficiency,
but struggle with aggregating local spatial representations, an essential
capability to SISR models. In response to this dichotomy, we propose to
harmonize the merits of both $3\times3$ and $1\times1$ kernels, and exploit a
great potential for lightweight SISR tasks. Specifically, we propose a simple
yet effective fully $1\times1$ convolutional network, named Shift-Conv-based
Network (SCNet). By incorporating a parameter-free spatial-shift operation, it
equips the fully $1\times1$ convolutional network with powerful representation
capability while impressive computational efficiency. Extensive experiments
demonstrate that SCNets, despite its fully $1\times1$ convolutional structure,
consistently matches or even surpasses the performance of existing lightweight
SR models that employ regular convolutions. The code and pre-trained models can
be found at https://github.com/Aitical/SCNet.
