  In recent years, deep learning has achieved great success in many natural
language processing tasks including named entity recognition. The shortcoming
is that a large amount of manually-annotated data is usually required. Previous
studies have demonstrated that active learning could elaborately reduce the
cost of data annotation, but there is still plenty of room for improvement. In
real applications we found existing uncertainty-based active learning
strategies have two shortcomings. Firstly, these strategies prefer to choose
long sequence explicitly or implicitly, which increase the annotation burden of
annotators. Secondly, some strategies need to invade the model and modify to
generate some additional information for sample selection, which will increase
the workload of the developer and increase the training/prediction time of the
model. In this paper, we first examine traditional active learning strategies
in a specific case of BiLstm-CRF that has widely used in named entity
recognition on several typical datasets. Then we propose an uncertainty-based
active learning strategy called Lowest Token Probability (LTP) which combines
the input and output of CRF to select informative instance. LTP is simple and
powerful strategy that does not favor long sequences and does not need to
invade the model. We test LTP on multiple datasets, and the experiments show
that LTP performs slightly better than traditional strategies with obviously
less annotation tokens on both sentence-level accuracy and entity-level
F1-score. Related code have been release on https://github.com/HIT-ICES/AL-NER
