  Despite the widespread adoption, there is a lack of research into how various
critical aspects of pretrained language models (PLMs) affect their performance
in hate speech detection. Through five research questions, our findings and
recommendations lay the groundwork for empirically investigating different
aspects of PLMs' use in hate speech detection. We deep dive into comparing
different pretrained models, evaluating their seed robustness, finetuning
settings, and the impact of pretraining data collection time. Our analysis
reveals early peaks for downstream tasks during pretraining, the limited
benefit of employing a more recent pretraining corpus, and the significance of
specific layers during finetuning. We further call into question the use of
domain-specific models and highlight the need for dynamic datasets for
benchmarking hate speech detection.
