  Transformers have been widely applied in text classification. Unfortunately,
real-world data contain anomalies and noisy labels that cause challenges for
state-of-art Transformers. This paper proposes Protoformer, a novel
self-learning framework for Transformers that can leverage problematic samples
for text classification. Protoformer features a selection mechanism for
embedding samples that allows us to efficiently extract and utilize anomalies
prototypes and difficult class prototypes. We demonstrated such capabilities on
datasets with diverse textual structures (e.g., Twitter, IMDB, ArXiv). We also
applied the framework to several models. The results indicate that Protoformer
can improve current Transformers in various empirical settings.
