  We consider the problem of task-agnostic feature upsampling in dense
prediction where an upsampling operator is required to facilitate both
region-sensitive tasks like semantic segmentation and detail-sensitive tasks
such as image matting. Existing upsampling operators often can work well in
either type of the tasks, but not both. In this work, we present FADE, a novel,
plug-and-play, and task-agnostic upsampling operator. FADE benefits from three
design choices: i) considering encoder and decoder features jointly in
upsampling kernel generation; ii) an efficient semi-shift convolutional
operator that enables granular control over how each feature point contributes
to upsampling kernels; iii) a decoder-dependent gating mechanism for enhanced
detail delineation. We first study the upsampling properties of FADE on toy
data and then evaluate it on large-scale semantic segmentation and image
matting. In particular, FADE reveals its effectiveness and task-agnostic
characteristic by consistently outperforming recent dynamic upsampling
operators in different tasks. It also generalizes well across convolutional and
transformer architectures with little computational overhead. Our work
additionally provides thoughtful insights on what makes for task-agnostic
upsampling. Code is available at: http://lnkiy.in/fade_in
