  Clinical performance audits are routinely performed in Emergency Medical
Services (EMS) to ensure adherence to treatment protocols, to identify
individual areas of weakness for remediation, and to discover systemic
deficiencies to guide the development of the training syllabus. At present,
these audits are performed by manual chart review which is time-consuming and
laborious. In this paper, we present an automatic audit system based on both
the structured and unstructured ambulance case records and clinical notes with
a deep neural network-based named entities recognition model. The dataset used
in this study contained 58,898 unlabelled ambulance incidents encountered by
the Singapore Civil Defence Force from 1st April 2019 to 30th June 2019. A
weakly-supervised training approach was adopted to label the sentences. Later
on, we trained three different models to perform the NER task. All three models
achieve F1 scores of around 0.981 under entity type matching evaluation and
around 0.976 under strict evaluation, while the BiLSTM-CRF model is 1~2 orders
of magnitude lighter and faster than our BERT-based models. Overall, our
approach yielded a named entity recognition model that could reliably identify
clinical entities from unstructured paramedic free-text reports. Our proposed
system may improve the efficiency of clinical performance audits and can also
help with EMS database research.
