  Existing CNN-based methods for pixel labeling heavily depend on multi-scale
features to meet the requirements of both semantic comprehension and detail
preservation. State-of-the-art pixel labeling neural networks widely exploit
conventional scale-transfer operations, i.e., up-sampling and down-sampling to
learn multi-scale features. In this work, we find that these operations lead to
scale-confused features and suboptimal performance because they are
spatial-invariant and directly transit all feature information cross scales
without spatial selection. To address this issue, we propose the Gated
Scale-Transfer Operation (GSTO) to properly transit spatial-filtered features
to another scale. Specifically, GSTO can work either with or without extra
supervision. Unsupervised GSTO is learned from the feature itself while the
supervised one is guided by the supervised probability matrix. Both forms of
GSTO are lightweight and plug-and-play, which can be flexibly integrated into
networks or modules for learning better multi-scale features. In particular, by
plugging GSTO into HRNet, we get a more powerful backbone (namely GSTO-HRNet)
for pixel labeling, and it achieves new state-of-the-art results on the COCO
benchmark for human pose estimation and other benchmarks for semantic
segmentation including Cityscapes, LIP and Pascal Context, with negligible
extra computational cost. Moreover, experiment results demonstrate that GSTO
can also significantly boost the performance of multi-scale feature aggregation
modules like PPM and ASPP. Code will be made available at
https://github.com/VDIGPKU/GSTO.
