  Systematic reviews, which entail the extraction of data from large numbers of
scientific documents, are an ideal avenue for the application of machine
learning. They are vital to many fields of science and philanthropy, but are
very time-consuming and require experts. Yet the three main stages of a
systematic review are easily done automatically: searching for documents can be
done via APIs and scrapers, selection of relevant documents can be done via
binary classification, and extraction of data can be done via
sequence-labelling classification. Despite the promise of automation for this
field, little research exists that examines the various ways to automate each
of these tasks. We construct a pipeline that automates each of these aspects,
and experiment with many human-time vs. system quality trade-offs. We test the
ability of classifiers to work well on small amounts of data and to generalise
to data from countries not represented in the training data. We test different
types of data extraction with varying difficulty in annotation, and five
different neural architectures to do the extraction. We find that we can get
surprising accuracy and generalisability of the whole pipeline system with only
2 weeks of human-expert annotation, which is only 15% of the time it takes to
do the whole review manually and can be repeated and extended to new data with
no additional effort.
