  Structural analysis methods (e.g., probing and feature attribution) are
increasingly important tools for neural network analysis. We propose a new
structural analysis method grounded in a formal theory of causal abstraction
that provides rich characterizations of model-internal representations and
their roles in input/output behavior. In this method, neural representations
are aligned with variables in interpretable causal models, and then interchange
interventions are used to experimentally verify that the neural representations
have the causal properties of their aligned variables. We apply this method in
a case study to analyze neural models trained on Multiply Quantified Natural
Language Inference (MQNLI) corpus, a highly complex NLI dataset that was
constructed with a tree-structured natural logic causal model. We discover that
a BERT-based model with state-of-the-art performance successfully realizes
parts of the natural logic model's causal structure, whereas a simpler baseline
model fails to show any such structure, demonstrating that BERT representations
encode the compositional structure of MQNLI.
