  Despite the growing demand for tuning foundation vision transformers (FViTs)
on downstream tasks, fully unleashing FViTs' potential under data-limited
scenarios (e.g., few-shot tuning) remains a challenge due to FViTs' data-hungry
nature. Common data augmentation techniques fall short in this context due to
the limited features contained in the few-shot tuning data. To tackle this
challenge, we first identify an opportunity for FViTs in few-shot tuning:
pretrained FViTs themselves have already learned highly representative features
from large-scale pretraining data, which are fully preserved during widely used
parameter-efficient tuning. We thus hypothesize that leveraging those learned
features to augment the tuning data can boost the effectiveness of few-shot
FViT tuning. To this end, we propose a framework called Hint-based Data
Augmentation (Hint-Aug), which aims to boost FViT in few-shot tuning by
augmenting the over-fitted parts of tuning samples with the learned features of
pretrained FViTs. Specifically, Hint-Aug integrates two key enablers: (1) an
Attentive Over-fitting Detector (AOD) to detect over-confident patches of
foundation ViTs for potentially alleviating their over-fitting on the few-shot
tuning data and (2) a Confusion-based Feature Infusion (CFI) module to infuse
easy-to-confuse features from the pretrained FViTs with the over-confident
patches detected by the above AOD in order to enhance the feature diversity
during tuning. Extensive experiments and ablation studies on five datasets and
three parameter-efficient tuning techniques consistently validate Hint-Aug's
effectiveness: 0.04% ~ 32.91% higher accuracy over the state-of-the-art (SOTA)
data augmentation method under various low-shot settings. For example, on the
Pet dataset, Hint-Aug achieves a 2.22% higher accuracy with 50% less training
data over SOTA data augmentation methods.
