  Currently, the most widespread neural network architecture for training
language models is the so called BERT which led to improvements in various
Natural Language Processing (NLP) tasks. In general, the larger the number of
parameters in a BERT model, the better the results obtained in these NLP tasks.
Unfortunately, the memory consumption and the training duration drastically
increases with the size of these models. In this article, we investigate
various training techniques of smaller BERT models: We combine different
methods from other BERT variants like ALBERT, RoBERTa, and relative positional
encoding. In addition, we propose two new fine-tuning modifications leading to
better performance: Class-Start-End tagging and a modified form of Linear Chain
Conditional Random Fields. Furthermore, we introduce Whole-Word Attention which
reduces BERTs memory usage and leads to a small increase in performance
compared to classical Multi-Head-Attention. We evaluate these techniques on
five public German Named Entity Recognition (NER) tasks of which two are
introduced by this article.
