  Several code summarization techniques have been proposed in the literature to
automatically document a code snippet or a function. Ideally, software
developers should be involved in assessing the quality of the generated
summaries. However, in most cases, researchers rely on automatic evaluation
metrics such as BLEU, ROUGE, and METEOR. These metrics are all based on the
same assumption: The higher the textual similarity between the generated
summary and a reference summary written by developers, the higher its quality.
However, there are two reasons for which this assumption falls short: (i)
reference summaries, e.g., code comments collected by mining software
repositories, may be of low quality or even outdated; (ii) generated summaries,
while using a different wording than a reference one, could be semantically
equivalent to it, thus still being suitable to document the code snippet. In
this paper, we perform a thorough empirical investigation on the
complementarity of different types of metrics in capturing the quality of a
generated summary. Also, we propose to address the limitations of existing
metrics by considering a new dimension, capturing the extent to which the
generated summary aligns with the semantics of the documented code snippet,
independently from the reference summary. To this end, we present a new metric
based on contrastive learning to capture said aspect. We empirically show that
the inclusion of this novel dimension enables a more effective representation
of developers' evaluations regarding the quality of automatically generated
summaries.
