  The ubiquity of offensive content on social media is a growing cause for
concern among companies and government organizations. Recently,
transformer-based models such as BERT, XLNET, and XLM-R have achieved
state-of-the-art performance in detecting various forms of offensive content
(e.g. hate speech, cyberbullying, and cyberaggression). However, the majority
of these models are limited in their capabilities due to their encoder-only
architecture, which restricts the number and types of labels in downstream
tasks. Addressing these limitations, this study presents the first pre-trained
model with encoder-decoder architecture for offensive language identification
with text-to-text transformers (T5) trained on two large offensive language
identification datasets; SOLID and CCTK. We investigate the effectiveness of
combining two datasets and selecting an optimal threshold in semi-supervised
instances in SOLID in the T5 retraining step. Our pre-trained T5 model
outperforms other transformer-based models fine-tuned for offensive language
detection, such as fBERT and HateBERT, in multiple English benchmarks.
Following a similar approach, we also train the first multilingual pre-trained
model for offensive language identification using mT5 and evaluate its
performance on a set of six different languages (German, Hindi, Korean,
Marathi, Sinhala, and Spanish). The results demonstrate that this multilingual
model achieves a new state-of-the-art on all the above datasets, showing its
usefulness in multilingual scenarios. Our proposed T5-based models will be made
freely available to the community.
