  The rapid growth of research publications has placed great demands on digital
libraries (DL) for advanced information management technologies. To cater to
these demands, techniques relying on knowledge-graph structures are being
advocated. In such graph-based pipelines, inferring semantic relations between
related scientific concepts is a crucial step. Recently, BERT-based pre-trained
models have been popularly explored for automatic relation classification.
Despite significant progress, most of them were evaluated in different
scenarios, which limits their comparability. Furthermore, existing methods are
primarily evaluated on clean texts, which ignores the digitization context of
early scholarly publications in terms of machine scanning and optical character
recognition (OCR). In such cases, the texts may contain OCR noise, in turn
creating uncertainty about existing classifiers' performances. To address these
limitations, we started by creating OCR-noisy texts based on three clean
corpora. Given these parallel corpora, we conducted a thorough empirical
evaluation of eight Bert-based classification models by focusing on three
factors: (1) Bert variants; (2) classification strategies; and, (3) OCR noise
impacts. Experiments on clean data show that the domain-specific pre-trained
Bert is the best variant to identify scientific relations. The strategy of
predicting a single relation each time outperforms the one simultaneously
identifying multiple relations in general. The optimal classifier's performance
can decline by around 10% to 20% in F-score on the noisy corpora. Insights
discussed in this study can help DL stakeholders select techniques for building
optimal knowledge-graph-based systems.
