  Spatial-wise dynamic convolution has become a promising approach to improving
the inference efficiency of deep networks. By allocating more computation to
the most informative pixels, such an adaptive inference paradigm reduces the
spatial redundancy in image features and saves a considerable amount of
unnecessary computation. However, the theoretical efficiency achieved by
previous methods can hardly translate into a realistic speedup, especially on
the multi-core processors (e.g. GPUs). The key challenge is that the existing
literature has only focused on designing algorithms with minimal computation,
ignoring the fact that the practical latency can also be influenced by
scheduling strategies and hardware properties. To bridge the gap between
theoretical computation and practical efficiency, we propose a latency-aware
spatial-wise dynamic network (LASNet), which performs coarse-grained spatially
adaptive inference under the guidance of a novel latency prediction model. The
latency prediction model can efficiently estimate the inference latency of
dynamic networks by simultaneously considering algorithms, scheduling
strategies, and hardware properties. We use the latency predictor to guide both
the algorithm design and the scheduling optimization on various hardware
platforms. Experiments on image classification, object detection and instance
segmentation demonstrate that the proposed framework significantly improves the
practical inference efficiency of deep networks. For example, the average
latency of a ResNet-101 on the ImageNet validation set could be reduced by 36%
and 46% on a server GPU (Nvidia Tesla-V100) and an edge device (Nvidia Jetson
TX2 GPU) respectively without sacrificing the accuracy. Code is available at
https://github.com/LeapLabTHU/LASNet.
