  Pooling layers are essential building blocks of convolutional neural networks
(CNNs), to reduce computational overhead and increase the receptive fields of
proceeding convolutional operations. Their goal is to produce downsampled
volumes that closely resemble the input volume while, ideally, also being
computationally and memory efficient. Meeting both these requirements remains a
challenge. To this end, we propose an adaptive and exponentially weighted
pooling method: adaPool. Our method learns a regional-specific fusion of two
sets of pooling kernels that are based on the exponent of the Dice-Sorensen
coefficient and the exponential maximum, respectively. AdaPool improves the
preservation of detail on a range of tasks including image and video
classification and object detection. A key property of adaPool is its
bidirectional nature. In contrast to common pooling methods, the learned
weights can also be used to upsample activation maps. We term this method
adaUnPool. We evaluate adaUnPool on image and video super-resolution and frame
interpolation. For benchmarking, we introduce Inter4K, a novel high-quality,
high frame-rate video dataset. Our experiments demonstrate that adaPool
systematically achieves better results across tasks and backbones, while
introducing a minor additional computational and memory overhead.
