  Pre-trained models for programming language have achieved dramatic empirical
improvements on a variety of code-related tasks such as code search, code
completion, code summarization, etc. However, existing pre-trained models
regard a code snippet as a sequence of tokens, while ignoring the inherent
structure of code, which provides crucial code semantics and would enhance the
code understanding process. We present GraphCodeBERT, a pre-trained model for
programming language that considers the inherent structure of code. Instead of
taking syntactic-level structure of code like abstract syntax tree (AST), we
use data flow in the pre-training stage, which is a semantic-level structure of
code that encodes the relation of "where-the-value-comes-from" between
variables. Such a semantic-level structure is neat and does not bring an
unnecessarily deep hierarchy of AST, the property of which makes the model more
efficient. We develop GraphCodeBERT based on Transformer. In addition to using
the task of masked language modeling, we introduce two structure-aware
pre-training tasks. One is to predict code structure edges, and the other is to
align representations between source code and code structure. We implement the
model in an efficient way with a graph-guided masked attention function to
incorporate the code structure. We evaluate our model on four tasks, including
code search, clone detection, code translation, and code refinement. Results
show that code structure and newly introduced pre-training tasks can improve
GraphCodeBERT and achieves state-of-the-art performance on the four downstream
tasks. We further show that the model prefers structure-level attentions over
token-level attentions in the task of code search.
