  Human annotations are an important source of information in the development
of natural language understanding approaches. As under the pressure of
productivity annotators can assign different labels to a given text, the
quality of produced annotations frequently varies. This is especially the case
if decisions are difficult, with high cognitive load, requires awareness of
broader context, or careful consideration of background knowledge. To alleviate
the problem, we propose two semi-supervised methods to guide the annotation
process: a Bayesian deep learning model and a Bayesian ensemble method. Using a
Bayesian deep learning method, we can discover annotations that cannot be
trusted and might require reannotation. A recently proposed Bayesian ensemble
method helps us to combine the annotators' labels with predictions of trained
models. According to the results obtained from three hate speech detection
experiments, the proposed Bayesian methods can improve the annotations and
prediction performance of BERT models.
