  The surge of interest in data augmentation within the realm of NLP has been
driven by the need to address challenges posed by hate speech domains, the
dynamic nature of social media vocabulary, and the demands for large-scale
neural networks requiring extensive training data. However, the prevalent use
of lexical substitution in data augmentation has raised concerns, as it may
inadvertently alter the intended meaning, thereby impacting the efficacy of
supervised machine learning models. In pursuit of suitable data augmentation
methods, this study explores both established legacy approaches and
contemporary practices such as Large Language Models (LLM), including GPT in
Hate Speech detection. Additionally, we propose an optimized utilization of
BERT-based encoder models with contextual cosine similarity filtration,
exposing significant limitations in prior synonym substitution methods. Our
comparative analysis encompasses five popular augmentation techniques: WordNet
and Fast-Text synonym replacement, Back-translation, BERT-mask contextual
augmentation, and LLM. Our analysis across five benchmarked datasets revealed
that while traditional methods like back-translation show low label alteration
rates (0.3-1.5%), and BERT-based contextual synonym replacement offers sentence
diversity but at the cost of higher label alteration rates (over 6%). Our
proposed BERT-based contextual cosine similarity filtration markedly reduced
label alteration to just 0.05%, demonstrating its efficacy in 0.7% higher F1
performance. However, augmenting data with GPT-3 not only avoided overfitting
with up to sevenfold data increase but also improved embedding space coverage
by 15% and classification F1 score by 1.4% over traditional methods, and by
0.8% over our method.
