  While transfer learning has become a ubiquitous technique used across Natural
Language Processing (NLP) tasks, it is often unable to replicate the
performance of pre-trained models on text of niche domains like Automotive. In
this paper we aim to understand the main characteristics of the distribution
shift with automotive domain text (describing technical functionalities such as
Cruise Control) and attempt to explain the potential reasons for the gap in
performance. We focus on performing the Named Entity Recognition (NER) task as
it requires strong lexical, syntactic and semantic understanding by the model.
Our experiments with 2 different encoders, namely BERT-Base-Uncased and
SciBERT-Base-Scivocab-Uncased have lead to interesting findings that showed: 1)
The performance of SciBERT is better than BERT when used for automotive domain,
2) Fine-tuning the language models with automotive domain text did not make
significant improvements to the NER performance, 3) The distribution shift is
challenging as it is characterized by lack of repeating contexts, sparseness of
entities, large number of Out-Of-Vocabulary (OOV) words and class overlap due
to domain specific nuances.
