  An accurate and substantial dataset is essential for training a reliable and
well-performing model. However, even manually annotated datasets contain label
errors, not to mention automatically labeled ones. Previous methods for label
denoising have primarily focused on detecting outliers and their permanent
removal - a process that is likely to over- or underfilter the dataset. In this
work, we propose AGRA: a new method for learning with noisy labels by using
Adaptive GRAdient-based outlier removal. Instead of cleaning the dataset prior
to model training, the dataset is dynamically adjusted during the training
process. By comparing the aggregated gradient of a batch of samples and an
individual example gradient, our method dynamically decides whether a
corresponding example is helpful for the model at this point or is
counter-productive and should be left out for the current update. Extensive
evaluation on several datasets demonstrates AGRA's effectiveness, while a
comprehensive results analysis supports our initial hypothesis: permanent hard
outlier removal is not always what model benefits the most from.
