  We present our HABERTOR model for detecting hatespeech in large scale
user-generated content. Inspired by the recent success of the BERT model, we
propose several modifications to BERT to enhance the performance on the
downstream hatespeech classification task. HABERTOR inherits BERT's
architecture, but is different in four aspects: (i) it generates its own
vocabularies and is pre-trained from the scratch using the largest scale
hatespeech dataset; (ii) it consists of Quaternion-based factorized components,
resulting in a much smaller number of parameters, faster training and
inferencing, as well as less memory usage; (iii) it uses our proposed
multi-source ensemble heads with a pooling layer for separate input sources, to
further enhance its effectiveness; and (iv) it uses a regularized adversarial
training with our proposed fine-grained and adaptive noise magnitude to enhance
its robustness. Through experiments on the large-scale real-world hatespeech
dataset with 1.4M annotated comments, we show that HABERTOR works better than
15 state-of-the-art hatespeech detection methods, including fine-tuning
Language Models. In particular, comparing with BERT, our HABERTOR is 4~5 times
faster in the training/inferencing phase, uses less than 1/3 of the memory, and
has better performance, even though we pre-train it by using less than 1% of
the number of words. Our generalizability analysis shows that HABERTOR
transfers well to other unseen hatespeech datasets and is a more efficient and
effective alternative to BERT for the hatespeech classification.
