  Much of software-engineering research relies on the naturalness of code, the
fact that code, in small code snippets, is repetitive and can be predicted
using statistical language models like n-gram. Although powerful, training such
models on large code corpus is tedious, time-consuming and sensitive to code
patterns (and practices) encountered during training. Consequently, these
models are often trained on a small corpora and estimate the language
naturalness that is relative to a specific style of programming or type of
project. To overcome these issues, we propose using pre-trained language models
to infer code naturalness. Pre-trained models are often built on big data, are
easy to use in an out-of-the-box way and include powerful learning associations
mechanisms. Our key idea is to quantify code naturalness through its
predictability, by using state-of-the-art generative pre-trained language
models. To this end, we infer naturalness by masking (omitting) code tokens,
one at a time, of code-sequences, and checking the models' ability to predict
them. To this end, we evaluate three different predictability metrics; a)
measuring the number of exact matches of the predictions, b) computing the
embedding similarity between the original and predicted code, i.e., similarity
at the vector space, and c) computing the confidence of the model when doing
the token completion task irrespective of the outcome. We implement this
workflow, named CodeBERT-nt, and evaluate its capability to prioritize buggy
lines over non-buggy ones when ranking code based on its naturalness. Our
results, on 2510 buggy versions of 40 projects from the SmartShark dataset,
show that CodeBERT-nt outperforms both, random-uniform and complexity-based
ranking techniques, and yields comparable results (slightly better) than the
n-gram models.
