  We show that learning affinity in upsampling provides an effective and
efficient approach to exploit pairwise interactions in deep networks.
Second-order features are commonly used in dense prediction to build adjacent
relations with a learnable module after upsampling such as non-local blocks.
Since upsampling is essential, learning affinity in upsampling can avoid
additional propagation layers, offering the potential for building compact
models. By looking at existing upsampling operators from a unified mathematical
perspective, we generalize them into a second-order form and introduce
Affinity-Aware Upsampling (A2U) where upsampling kernels are generated using a
light-weight lowrank bilinear model and are conditioned on second-order
features. Our upsampling operator can also be extended to downsampling. We
discuss alternative implementations of A2U and verify their effectiveness on
two detail-sensitive tasks: image reconstruction on a toy dataset; and a
largescale image matting task where affinity-based ideas constitute mainstream
matting approaches. In particular, results on the Composition-1k matting
dataset show that A2U achieves a 14% relative improvement in the SAD metric
against a strong baseline with negligible increase of parameters (<0.5%).
Compared with the state-of-the-art matting network, we achieve 8% higher
performance with only 40% model complexity.
