  Although transformer-based Neural Language Models demonstrate impressive
performance on a variety of tasks, their generalization abilities are not well
understood. They have been shown to perform strongly on subject-verb number
agreement in a wide array of settings, suggesting that they learned to track
syntactic dependencies during their training even without explicit supervision.
In this paper, we examine the extent to which BERT is able to perform
lexically-independent subject-verb number agreement (NA) on targeted syntactic
templates. To do so, we disrupt the lexical patterns found in naturally
occurring stimuli for each targeted structure in a novel fine-grained analysis
of BERT's behavior. Our results on nonce sentences suggest that the model
generalizes well for simple templates, but fails to perform
lexically-independent syntactic generalization when as little as one attractor
is present.
