  Inspired by cognitive neuroscience studies, we introduce a novel `decoding
probing' method that uses minimal pairs benchmark (BLiMP) to probe internal
linguistic characteristics in neural language models layer by layer. By
treating the language model as the `brain' and its representations as `neural
activations', we decode grammaticality labels of minimal pairs from the
intermediate layers' representations. This approach reveals: 1) Self-supervised
language models capture abstract linguistic structures in intermediate layers
that GloVe and RNN language models cannot learn. 2) Information about syntactic
grammaticality is robustly captured through the first third layers of GPT-2 and
also distributed in later layers. As sentence complexity increases, more layers
are required for learning grammatical capabilities. 3) Morphological and
semantics/syntax interface-related features are harder to capture than syntax.
4) For Transformer-based models, both embeddings and attentions capture
grammatical features but show distinct patterns. Different attention heads
exhibit similar tendencies toward various linguistic phenomena, but with varied
contributions.
