  Research has shown that neural models implicitly encode linguistic features,
but there has been no research showing \emph{how} these encodings arise as the
models are trained. We present the first study on the learning dynamics of
neural language models, using a simple and flexible analysis method called
Singular Vector Canonical Correlation Analysis (SVCCA), which enables us to
compare learned representations across time and across models, without the need
to evaluate directly on annotated data. We probe the evolution of syntactic,
semantic, and topic representations and find that part-of-speech is learned
earlier than topic; that recurrent layers become more similar to those of a
tagger during training; and embedding layers less similar. Our results and
methods could inform better learning algorithms for NLP models, possibly to
incorporate linguistic information more effectively.
