  National research evaluation initiatives and incentive schemes have
previously chosen between simplistic quantitative indicators and time-consuming
peer review, sometimes supported by bibliometrics. Here we assess whether
artificial intelligence (AI) could provide a third alternative, estimating
article quality using more multiple bibliometric and metadata inputs. We
investigated this using provisional three-level REF2021 peer review scores for
84,966 articles submitted to the UK Research Excellence Framework 2021,
matching a Scopus record 2014-18 and with a substantial abstract. We found that
accuracy is highest in the medical and physical sciences Units of Assessment
(UoAs) and economics, reaching 42% above the baseline (72% overall) in the best
case. This is based on 1000 bibliometric inputs and half of the articles used
for training in each UoA. Prediction accuracies above the baseline for the
social science, mathematics, engineering, arts, and humanities UoAs were much
lower or close to zero. The Random Forest Classifier (standard or ordinal) and
Extreme Gradient Boosting Classifier algorithms performed best from the 32
tested. Accuracy was lower if UoAs were merged or replaced by Scopus broad
categories. We increased accuracy with an active learning strategy and by
selecting articles with higher prediction probabilities, as estimated by the
algorithms, but this substantially reduced the number of scores predicted.
