  Technology-assisted review (TAR) refers to iterative active learning
workflows for document review in high recall retrieval (HRR) tasks. TAR
research and most commercial TAR software have applied linear models such as
logistic regression to lexical features. Transformer-based models with
supervised tuning are known to improve effectiveness on many text
classification tasks, suggesting their use in TAR. We indeed find that the
pre-trained BERT model reduces review cost by 10% to 15% in TAR workflows
simulated on the RCV1-v2 newswire collection. In contrast, we likewise
determined that linear models outperform BERT for simulated legal discovery
topics on the Jeb Bush e-mail collection. This suggests the match between
transformer pre-training corpora and the task domain is of greater significance
than generally appreciated. Additionally, we show that just-right language
model fine-tuning on the task collection before starting active learning is
critical. Too little or too much fine-tuning hinders performance, worse than
that of linear models, even for a favorable corpus such as RCV1-v2.
