  Statistical language modeling techniques have successfully been applied to
source code, yielding a variety of new software development tools, such as
tools for code suggestion and improving readability. A major issue with these
techniques is that code introduces new vocabulary at a far higher rate than
natural language, as new identifier names proliferate. But traditional language
models limit the vocabulary to a fixed set of common words. For code, this
strong assumption has been shown to have a significant negative effect on
predictive performance. But the open vocabulary version of the neural network
language models for code have not been introduced in the literature. We present
a new open-vocabulary neural language model for code that is not limited to a
fixed vocabulary of identifier names. We employ a segmentation into subword
units, subsequences of tokens chosen based on a compression criterion,
following previous work in machine translation. Our network achieves best in
class performance, outperforming even the state-of-the-art methods of
Hellendoorn and Devanbu that are designed specifically to model code.
Furthermore, we present a simple method for dynamically adapting the model to a
new test project, resulting in increased performance. We showcase our
methodology on code corpora in three different languages of over a billion
tokens each, hundreds of times larger than in previous work. To our knowledge,
this is the largest neural language model for code that has been reported.
