  In the feature maps of CNNs, there commonly exists considerable spatial
redundancy that leads to much repetitive processing. Towards reducing this
superfluous computation, we propose to compute features only at sparsely
sampled locations, which are probabilistically chosen according to activation
responses, and then densely reconstruct the feature map with an efficient
interpolation procedure. With this sampling-interpolation scheme, our network
avoids expending computation on spatial locations that can be effectively
interpolated, while being robust to activation prediction errors through
broadly distributed sampling. A technical challenge of this sampling-based
approach is that the binary decision variables for representing discrete
sampling locations are non-differentiable, making them incompatible with
backpropagation. To circumvent this issue, we make use of a reparameterization
trick based on the Gumbel-Softmax distribution, with which backpropagation can
iterate these variables towards binary values. The presented network is
experimentally shown to save substantial computation while maintaining accuracy
over a variety of computer vision tasks.
