  In this paper we address the challenge of extracting scientific references
from patents. We approach the problem as a sequence labelling task and
investigate the merits of BERT models to the extraction of these long
sequences. References in patents to scientific literature are relevant to study
the connection between science and industry. Most prior work only uses the
front-page citations for this analysis, which are provided in the metadata of
patent archives. In this paper we build on prior work using Conditional Random
Fields (CRF) and Flair for reference extraction. We improve the quality of the
training data and train three BERT-based models on the labelled data (BERT,
bioBERT, sciBERT). We find that the improved training data leads to a large
improvement in the quality of the trained models. In addition, the BERT models
beat CRF and Flair, with recall scores around 97% obtained with cross
validation. With the best model we label a large collection of 33 thousand
patents, extract the citations, and match them to publications in the Web of
Science database. We extract 50% more references than with the old training
data and methods: 735 thousand references in total. With these
patent-publication links, follow-up research will further analyze which types
of scientific work lead to inventions.
