  This paper presents a procedure for and evaluation of using a semantic
similarity metric as a loss function for neural source code summarization. Code
summarization is the task of writing natural language descriptions of source
code. Neural code summarization refers to automated techniques for generating
these descriptions using neural networks. Almost all current approaches involve
neural networks as either standalone models or as part of a pretrained large
language models e.g., GPT, Codex, LLaMA. Yet almost all also use a categorical
cross-entropy (CCE) loss function for network optimization. Two problems with
CCE are that 1) it computes loss over each word prediction one-at-a-time,
rather than evaluating a whole sentence, and 2) it requires a perfect
prediction, leaving no room for partial credit for synonyms. In this paper, we
extend our previous work on semantic similarity metrics to show a procedure for
using semantic similarity as a loss function to alleviate this problem, and we
evaluate this procedure in several settings in both metrics-driven and human
studies. In essence, we propose to use a semantic similarity metric to
calculate loss over the whole output sentence prediction per training batch,
rather than just loss for each word. We also propose to combine our loss with
CCE for each word, which streamlines the training process compared to
baselines. We evaluate our approach over several baselines and report
improvement in the vast majority of conditions.
