  Given the large-scale data and the high annotation cost,
pretraining-finetuning becomes a popular paradigm in multiple computer vision
tasks. Previous research has covered both the unsupervised pretraining and
supervised finetuning in this paradigm, while little attention is paid to
exploiting the annotation budget for finetuning. To fill in this gap, we
formally define this new active finetuning task focusing on the selection of
samples for annotation in the pretraining-finetuning paradigm. We propose a
novel method called ActiveFT for active finetuning task to select a subset of
data distributing similarly with the entire unlabeled pool and maintaining
enough diversity by optimizing a parametric model in the continuous space. We
prove that the Earth Mover's distance between the distributions of the selected
subset and the entire data pool is also reduced in this process. Extensive
experiments show the leading performance and high efficiency of ActiveFT
superior to baselines on both image classification and semantic segmentation.
Our code is released at https://github.com/yichen928/ActiveFT.
