  Large language models (LLMs) such as Llama 2 perform very well on tasks that
involve both natural language and source code, particularly code summarization
and code generation. We show that for the task of code summarization, the
performance of these models on individual examples often depends on the amount
of (subword) token overlap between the code and the corresponding reference
natural language descriptions in the dataset. This token overlap arises because
the reference descriptions in standard datasets (corresponding to docstrings in
large code bases) are often highly similar to the names of the functions they
describe. We also show that this token overlap occurs largely in the function
names of the code and compare the relative performance of these models after
removing function names versus removing code structure. We also show that using
multiple evaluation metrics like BLEU and BERTScore gives us very little
additional insight since these metrics are highly correlated with each other.
