  Multi-scale learning frameworks have been regarded as a capable class of
models to boost semantic segmentation. The problem nevertheless is not trivial
especially for the real-world deployments, which often demand high efficiency
in inference latency. In this paper, we thoroughly analyze the design of
convolutional blocks (the type of convolutions and the number of channels in
convolutions), and the ways of interactions across multiple scales, all from
lightweight standpoint for semantic segmentation. With such in-depth
comparisons, we conclude three principles, and accordingly devise Lightweight
and Progressively-Scalable Networks (LPS-Net) that novelly expands the network
complexity in a greedy manner. Technically, LPS-Net first capitalizes on the
principles to build a tiny network. Then, LPS-Net progressively scales the tiny
network to larger ones by expanding a single dimension (the number of
convolutional blocks, the number of channels, or the input resolution) at one
time to meet the best speed/accuracy tradeoff. Extensive experiments conducted
on three datasets consistently demonstrate the superiority of LPS-Net over
several efficient semantic segmentation methods. More remarkably, our LPS-Net
achieves 73.4% mIoU on Cityscapes test set, with the speed of 413.5FPS on an
NVIDIA GTX 1080Ti, leading to a performance improvement by 1.5% and a 65%
speed-up against the state-of-the-art STDC. Code is available at
\url{https://github.com/YihengZhang-CV/LPS-Net}.
