  In recent years, the use of deep learning in language models gained much
attention. Some research projects claim that they can generate text that can be
interpreted as human-writing, enabling new possibilities in many application
areas. Among the different areas related to language processing, one of the
most notable in applying this type of modeling is programming languages. For
years, the Machine Learning community has been researching this software
engineering area, pursuing goals like applying different approaches to
auto-complete, generate, fix, or evaluate code programmed by humans.
Considering the increasing popularity of the Deep-Learning-enabled language
models approach, we detected a lack of empirical papers that compare different
deep learning architectures to create and use language models based on
programming code. This paper compares different neural network architectures
like AWD-LSTMs, AWD-QRNNs, and Transformer while using transfer learning and
different tokenizations to see how they behave in building language models
using a Python dataset for code generation and filling mask tasks. Considering
the results, we discuss each approach's different strengths and weaknesses and
what gaps we find to evaluate the language models or apply them in a real
programming context.
