  Initially developed for natural language processing (NLP), Transformers are
now widely used for source code processing, due to the format similarity
between source code and text. In contrast to natural language, source code is
strictly structured, i.e., it follows the syntax of the programming language.
Several recent works develop Transformer modifications for capturing syntactic
information in source code. The drawback of these works is that they do not
compare to each other and consider different tasks. In this work, we conduct a
thorough empirical study of the capabilities of Transformers to utilize
syntactic information in different tasks. We consider three tasks (code
completion, function naming and bug fixing) and re-implement different
syntax-capturing modifications in a unified framework. We show that
Transformers are able to make meaningful predictions based purely on syntactic
information and underline the best practices of taking the syntactic
information into account for improving the performance of the model.
