  Pathology reports contain useful information such as the main involved organ,
diagnosis, etc. These information can be identified from the free text reports
and used for large-scale statistical analysis or serve as annotation for other
modalities such as pathology slides images. However, manual classification for
a huge number of reports on multiple tasks is labor-intensive. In this paper,
we have developed an automatic text classifier based on BERT and we propose a
human-centric metric to evaluate the model. According to the model confidence,
we identify low-confidence cases that require further expert annotation and
high-confidence cases that are automatically classified. We report the
percentage of low-confidence cases and the performance of automatically
classified cases. On the high-confidence cases, the model achieves
classification accuracy comparable to pathologists. This leads a potential of
reducing 80% to 98% of the manual annotation workload.
