  In neural network models of language, words are commonly represented using
context-invariant representations (word embeddings) which are then put in
context in the hidden layers. Since words are often ambiguous, representing the
contextually relevant information is not trivial. We investigate how an LSTM
language model deals with lexical ambiguity in English, designing a method to
probe its hidden representations for lexical and contextual information about
words. We find that both types of information are represented to a large
extent, but also that there is room for improvement for contextual information.
