  In recent years, there has been a wide interest in designing deep neural
network-based models that automate downstream software engineering tasks on
source code, such as code document generation, code search, and program repair.
Although the main objective of these studies is to improve the effectiveness of
the downstream task, many studies only attempt to employ the next best neural
network model, without a proper in-depth analysis of why a particular solution
works or does not, on particular tasks or scenarios. In this paper, using an
example eXplainable AI (XAI) method (attention mechanism), we study two recent
large language models (LLMs) for code (CodeBERT and GraphCodeBERT) on a set of
software engineering downstream tasks: code document generation (CDG), code
refinement (CR), and code translation (CT). Through quantitative and
qualitative studies, we identify what CodeBERT and GraphCodeBERT learn (put the
highest attention on, in terms of source code token types), on these tasks. We
also show some of the common patterns when the model does not work as expected
(performs poorly even on easy problems) and suggest recommendations that may
alleviate the observed challenges.
