  We present a simple yet effective method to train a named entity recognition
(NER) model that operates on business telephone conversation transcripts that
contain noise due to the nature of spoken conversation and artifacts of
automatic speech recognition. We first fine-tune LUKE, a state-of-the-art Named
Entity Recognition (NER) model, on a limited amount of transcripts, then use it
as the teacher model to teach a smaller DistilBERT-based student model using a
large amount of weakly labeled data and a small amount of human-annotated data.
The model achieves high accuracy while also satisfying the practical
constraints for inclusion in a commercial telephony product: realtime
performance when deployed on cost-effective CPUs rather than GPUs.
