  Modern convolutional neural networks apply the same operations on every pixel
in an image. However, not all image regions are equally important. To address
this inefficiency, we propose a method to dynamically apply convolutions
conditioned on the input image. We introduce a residual block where a small
gating branch learns which spatial positions should be evaluated. These
discrete gating decisions are trained end-to-end using the Gumbel-Softmax
trick, in combination with a sparsity criterion. Our experiments on CIFAR,
ImageNet and MPII show that our method has better focus on the region of
interest and better accuracy than existing methods, at a lower computational
complexity. Moreover, we provide an efficient CUDA implementation of our
dynamic convolutions using a gather-scatter approach, achieving a significant
improvement in inference speed with MobileNetV2 residual blocks. On human pose
estimation, a task that is inherently spatially sparse, the processing speed is
increased by 60% with no loss in accuracy.
