  Code pre-trained models (CodePTMs) have recently demonstrated significant
success in code intelligence. To interpret these models, some probing methods
have been applied. However, these methods fail to consider the inherent
characteristics of codes. In this paper, to address the problem, we propose a
novel probing method CAT-probing to quantitatively interpret how CodePTMs
attend code structure. We first denoise the input code sequences based on the
token types pre-defined by the compilers to filter those tokens whose attention
scores are too small. After that, we define a new metric CAT-score to measure
the commonality between the token-level attention scores generated in CodePTMs
and the pair-wise distances between corresponding AST nodes. The higher the
CAT-score, the stronger the ability of CodePTMs to capture code structure. We
conduct extensive experiments to integrate CAT-probing with representative
CodePTMs for different programming languages. Experimental results show the
effectiveness of CAT-probing in CodePTM interpretation. Our codes and data are
publicly available at https://github.com/nchen909/CodeAttention.
