  Recurrent neural networks (RNNs) have reached striking performance in many
natural language processing tasks. This has renewed interest in whether these
generic sequence processing devices are inducing genuine linguistic knowledge.
Nearly all current analytical studies, however, initialize the RNNs with a
vocabulary of known words, and feed them tokenized input during training. We
present a multi-lingual study of the linguistic knowledge encoded in RNNs
trained as character-level language models, on input data with word boundaries
removed. These networks face a tougher and more cognitively realistic task,
having to discover any useful linguistic unit from scratch based on input
statistics. The results show that our "near tabula rasa" RNNs are mostly able
to solve morphological, syntactic and semantic tasks that intuitively
presuppose word-level knowledge, and indeed they learned, to some extent, to
track word boundaries. Our study opens the door to speculations about the
necessity of an explicit, rigid word lexicon in language learning and usage.
