  Pre-trained language models have demonstrated impressive performance in both
natural language processing and program understanding, which represent the
input as a token sequence without explicitly modeling its structure. Some prior
works show that pre-trained language models can capture the syntactic rules of
natural languages without finetuning on syntax understanding tasks. However,
there is limited understanding of how well pre-trained models understand the
code structure so far. In this work, we perform the first thorough benchmarking
of the state-of-the-art pre-trained models for identifying the syntactic
structures of programs. Specifically, we introduce CodeSyntax, a large-scale
dataset of programs annotated with the syntactic relationships in their
corresponding abstract syntax trees. Our key observation is that existing
language models pretrained on code still lack the understanding of code syntax.
In fact, these pre-trained programming language models fail to match the
performance of simple baselines based on positional offsets and keywords. We
also present a natural language benchmark to highlight the differences between
natural languages and programming languages in terms of syntactic structure
understanding. Our findings point out key limitations of existing pre-training
methods for programming languages, and suggest the importance of modeling code
syntactic structures.
