  Runtime and memory consumption are two important aspects for efficient image
super-resolution (EISR) models to be deployed on resource-constrained devices.
Recent advances in EISR exploit distillation and aggregation strategies with
plenty of channel split and concatenation operations to make full use of
limited hierarchical features. In contrast, sequential network operations avoid
frequently accessing preceding states and extra nodes, and thus are beneficial
to reducing the memory consumption and runtime overhead. Following this idea,
we design our lightweight network backbone by mainly stacking multiple highly
optimized convolution and activation layers and decreasing the usage of feature
fusion. We propose a novel sequential attention branch, where every pixel is
assigned an important factor according to local and global contexts, to enhance
high-frequency details. In addition, we tailor the residual block for EISR and
propose an enhanced residual block (ERB) to further accelerate the network
inference. Finally, combining all the above techniques, we construct a fast and
memory-efficient network (FMEN) and its small version FMEN-S, which runs 33%
faster and reduces 74% memory consumption compared with the state-of-the-art
EISR model: E-RFDN, the champion in AIM 2020 efficient super-resolution
challenge. Besides, FMEN-S achieves the lowest memory consumption and the
second shortest runtime in NTIRE 2022 challenge on efficient super-resolution.
Code is available at https://github.com/NJU-Jet/FMEN.
