  Modern deep learning systems are data-hungry. Learning with web data is one
of the feasible solutions, but will introduce label noise inevitably, which can
hinder the performance of deep neural networks. Sample selection is an
effective way to deal with label noise. The key is to separate clean samples
based on some criterion. Previous methods pay more attention to the small loss
criterion where small-loss samples are regarded as clean ones. Nevertheless,
such a strategy relies on the learning dynamics of each data instance. Some
noisy samples are still memorized due to frequently occurring corrupted
learning patterns. To tackle this problem, a training-free surrogate model is
preferred, freeing from the effect of memorization. In this work, we propose to
leverage the vision-language surrogate model CLIP to filter noisy samples
automatically. CLIP brings external knowledge to facilitate the selection of
clean samples with its ability of text-image alignment. Furthermore, a margin
adaptive loss is designed to regularize the selection bias introduced by CLIP,
providing robustness to label noise. We validate the effectiveness of our
proposed method on both real-world and synthetic noisy datasets. Our method
achieves significant improvement without CLIP involved during the inference
stage.
