  This paper introduces Filtered Corpus Training, a method that trains language
models (LMs) on corpora with certain linguistic constructions filtered out from
the training data, and uses it to measure the ability of LMs to perform
linguistic generalization on the basis of indirect evidence. We apply the
method to both LSTM and Transformer LMs (of roughly comparable size),
developing filtered corpora that target a wide range of linguistic phenomena.
Our results show that while transformers are better qua LMs (as measured by
perplexity), both models perform equally and surprisingly well on linguistic
generalization measures, suggesting that they are capable of generalizing from
indirect evidence.
