  Language models such as RNN, LSTM or other variants have been widely used as
generative models in natural language processing. In last few years, taking
source code as natural languages, parsing source code into a token sequence and
using a language model such as LSTM to train that sequence are state-of-art
methods to get a generative model for solving the problem of code completion.
However, for source code with hundreds of statements, traditional LSTM model or
attention-based LSTM model failed to capture the long term dependency of source
code. In this paper, we propose a novel statement-level language model (SLM)
which uses BiLSTM to generate the embedding for each statement. The standard
LSTM is adopted in SLM to iterate and accumulate the embedding of each
statement in context to help predict next code. The statement level attention
mechanism is also adopted in the model. The proposed model SLM is aimed at
token level code completion. The experiments on inner-project and cross-project
data sets indicate that the newly proposed statement-level language model with
attention mechanism (SLM) outperforms all other state-of-art models in token
level code completion task.
