  Test-time task adaptation in few-shot learning aims to adapt a pre-trained
task-agnostic model for capturing taskspecific knowledge of the test task, rely
only on few-labeled support samples. Previous approaches generally focus on
developing advanced algorithms to achieve the goal, while neglecting the
inherent problems of the given support samples. In fact, with only a handful of
samples available, the adverse effect of either the image noise (a.k.a.
X-noise) or the label noise (a.k.a. Y-noise) from support samples can be
severely amplified. To address this challenge, in this work we propose DEnoised
Task Adaptation (DETA), a first, unified image- and label-denoising framework
orthogonal to existing task adaptation approaches. Without extra supervision,
DETA filters out task-irrelevant, noisy representations by taking advantage of
both global visual information and local region details of support samples. On
the challenging Meta-Dataset, DETA consistently improves the performance of a
broad spectrum of baseline methods applied on various pre-trained models.
Notably, by tackling the overlooked image noise in Meta-Dataset, DETA
establishes new state-of-the-art results. Code is released at
https://github.com/JimZAI/DETA.
