  Label noise remains a challenge for training robust classification models.
Most methods for mitigating label noise have been benchmarked using primarily
datasets with synthetic noise. While the need for datasets with realistic noise
distribution has partially been addressed by web-scraped benchmarks such as
WebVision and Clothing1M, those benchmarks are restricted to the computer
vision domain. With the growing importance of Transformer-based models, it is
crucial to establish text classification benchmarks for learning with noisy
labels. In this paper, we present AlleNoise, a new curated text classification
benchmark dataset with real-world instance-dependent label noise, containing
over 500,000 examples across approximately 5,600 classes, complemented with a
meaningful, hierarchical taxonomy of categories. The noise distribution comes
from actual users of a major e-commerce marketplace, so it realistically
reflects the semantics of human mistakes. In addition to the noisy labels, we
provide human-verified clean labels, which help to get a deeper insight into
the noise distribution, unlike web-scraped datasets typically used in the
field. We demonstrate that a representative selection of established methods
for learning with noisy labels is inadequate to handle such real-world noise.
In addition, we show evidence that these algorithms do not alleviate excessive
memorization. As such, with AlleNoise, we set the bar high for the development
of label noise methods that can handle real-world label noise in text
classification tasks. The code and dataset are available for download at
https://github.com/allegro/AlleNoise.
