  Character language models have access to surface morphological patterns, but
it is not clear whether or how they learn abstract morphological regularities.
We instrument a character language model with several probes, finding that it
can develop a specific unit to identify word boundaries and, by extension,
morpheme boundaries, which allows it to capture linguistic properties and
regularities of these units. Our language model proves surprisingly good at
identifying the selectional restrictions of English derivational morphemes, a
task that requires both morphological and syntactic awareness. Thus we conclude
that, when morphemes overlap extensively with the words of a language, a
character language model can perform morphological abstraction.
