  We show how the spellings of known words can help us deal with unknown words
in open-vocabulary NLP tasks. The method we propose can be used to extend any
closed-vocabulary generative model, but in this paper we specifically consider
the case of neural language modeling. Our Bayesian generative story combines a
standard RNN language model (generating the word tokens in each sentence) with
an RNN-based spelling model (generating the letters in each word type). These
two RNNs respectively capture sentence structure and word structure, and are
kept separate as in linguistics. By invoking the second RNN to generate
spellings for novel words in context, we obtain an open-vocabulary language
model. For known words, embeddings are naturally inferred by combining evidence
from type spelling and token context. Comparing to baselines (including a novel
strong baseline), we beat previous work and establish state-of-the-art results
on multiple datasets.
