  Deep architectures such as Transformers are sometimes criticized for having
uninterpretable "black-box" representations. We use causal intervention
analysis to show that, in fact, some linguistic features are represented in a
linear, interpretable format. Specifically, we show that BERT's ability to
conjugate verbs relies on a linear encoding of subject number that can be
manipulated with predictable effects on conjugation accuracy. This encoding is
found in the subject position at the first layer and the verb position at the
last layer, but distributed across positions at middle layers, particularly
when there are multiple cues to subject number.
