  Programming-based Pre-trained Language Models (PPLMs) such as CodeBERT have
achieved great success in many downstream code-related tasks. Since the memory
and computational complexity of self-attention in the Transformer grow
quadratically with the sequence length, PPLMs typically limit the code length
to 512. However, codes in real-world applications are generally long, such as
code searches, which cannot be processed efficiently by existing PPLMs. To
solve this problem, in this paper, we present SASA, a Structure-Aware Sparse
Attention mechanism, which reduces the complexity and improves performance for
long code understanding tasks. The key components in SASA are top-$k$ sparse
attention and Abstract Syntax Tree (AST)-based structure-aware attention. With
top-$k$ sparse attention, the most crucial attention relation can be obtained
with a lower computational cost. As the code structure represents the logic of
the code statements, which is a complement to the code sequence
characteristics, we further introduce AST structures into attention. Extensive
experiments on CodeXGLUE tasks show that SASA achieves better performance than
the competing baselines.
