  Many convolutional neural networks (CNNs) rely on progressive downsampling of
their feature maps to increase the network's receptive field and decrease
computational cost. However, this comes at the price of losing granularity in
the feature maps, limiting the ability to correctly understand images or
recover fine detail in dense prediction tasks. To address this, common practice
is to replace the last few downsampling operations in a CNN with dilated
convolutions, allowing to retain the feature map resolution without reducing
the receptive field, albeit increasing the computational cost. This allows to
trade off predictive performance against cost, depending on the output feature
resolution. By either regularly downsampling or not downsampling the entire
feature map, existing work implicitly treats all regions of the input image and
subsequent feature maps as equally important, which generally does not hold. We
propose an adaptive downsampling scheme that generalizes the above idea by
allowing to process informative regions at a higher resolution than less
informative ones. In a variety of experiments, we demonstrate the versatility
of our adaptive downsampling strategy and empirically show that it improves the
cost-accuracy trade-off of various established CNNs.
