  It has been found that software, like natural language texts, exhibits
"naturalness", which can be captured by statistical language models. In recent
years, neural language models have been proposed to represent the naturalness
of software through deep learning. In this paper, we conduct an experimental
evaluation of state-of-the-art neural language models for source code,
including RNN-based models and Transformer-XL based models. Through experiments
on a large-scale Python code corpus, we find that the Transformer-XL model
outperforms RNN-based models (including LSTM and GRU models) in capturing the
naturalness of software, with far less computational cost.
