  We introduce a linguistically enhanced combination of pre-training methods
for transformers. The pre-training objectives include POS-tagging, synset
prediction based on semantic knowledge graphs, and parent prediction based on
dependency parse trees. Our approach achieves competitive results on the
Natural Language Inference task, compared to the state of the art. Specifically
for smaller models, the method results in a significant performance boost,
emphasizing the fact that intelligent pre-training can make up for fewer
parameters and help building more efficient models. Combining POS-tagging and
synset prediction yields the overall best results.
