  A large number of studies that analyze deep neural network models and their
ability to encode various linguistic and non-linguistic concepts provide an
interpretation of the inner mechanics of these models. The scope of the
analyses is limited to pre-defined concepts that reinforce the traditional
linguistic knowledge and do not reflect on how novel concepts are learned by
the model. We address this limitation by discovering and analyzing latent
concepts learned in neural network models in an unsupervised fashion and
provide interpretations from the model's perspective. In this work, we study:
i) what latent concepts exist in the pre-trained BERT model, ii) how the
discovered latent concepts align or diverge from classical linguistic hierarchy
and iii) how the latent concepts evolve across layers. Our findings show: i) a
model learns novel concepts (e.g. animal categories and demographic groups),
which do not strictly adhere to any pre-defined categorization (e.g. POS,
semantic tags), ii) several latent concepts are based on multiple properties
which may include semantics, syntax, and morphology, iii) the lower layers in
the model dominate in learning shallow lexical concepts while the higher layers
learn semantic relations and iv) the discovered latent concepts highlight
potential biases learned in the model. We also release a novel BERT ConceptNet
dataset (BCN) consisting of 174 concept labels and 1M annotated instances.
