  Most Named Entity Recognition (NER) models operate under the assumption that
training datasets are fully labelled. While it is valid for established
datasets like CoNLL 2003 and OntoNotes, sometimes it is not feasible to obtain
the complete dataset annotation. These situations may occur, for instance,
after selective annotation of entities for cost reduction. This work presents
an approach to finetuning BERT on such partially labelled datasets using
self-supervision and label preprocessing. Our approach outperforms the previous
LSTM-based label preprocessing baseline, significantly improving the
performance on poorly labelled datasets. We demonstrate that following our
approach while finetuning RoBERTa on CoNLL 2003 dataset with only 10% of total
entities labelled is enough to reach the performance of the baseline trained on
the same dataset with 50% of the entities labelled.
