  In the process of Systematic Literature Review, citation screening is
estimated to be one of the most time-consuming steps. Multiple approaches to
automate it using various machine learning techniques have been proposed. The
first research papers that apply deep neural networks to this problem were
published in the last two years. In this work, we conduct a replicability study
of the first two deep learning papers for citation screening and evaluate their
performance on 23 publicly available datasets. While we succeeded in
replicating the results of one of the papers, we were unable to replicate the
results of the other. We summarise the challenges involved in the replication,
including difficulties in obtaining the datasets to match the experimental
setup of the original papers and problems with executing the original source
code. Motivated by this experience, we subsequently present a simpler model
based on averaging word embeddings that outperforms one of the models on 18 out
of 23 datasets and is, on average, 72 times faster than the second replicated
approach. Finally, we measure the training time and the invariance of the
models when exposed to a variety of input features and random initialisations,
demonstrating differences in the robustness of these approaches.
