  The task of code generation from natural language (NL2Code) has become
extremely popular, especially with the advent of Large Language Models (LLMs).
However, efforts to quantify and track this progress have suffered due to a
lack of reliable metrics for functional correctness. While popular benchmarks
like HumanEval have test cases to enable reliable evaluation of correctness, it
is time-consuming and requires human effort to collect test cases. As an
alternative several reference-based evaluation metrics have been proposed, with
embedding-based metrics like CodeBERTScore being touted as having a high
correlation with human preferences and functional correctness. In our work, we
analyze the ability of embedding-based metrics like CodeBERTScore to measure
functional correctness and other helpful constructs like editing effort by
analyzing outputs of ten models over two popular code generation benchmarks.
Our results show that while they have a weak correlation with functional
correctness (0.16), they are strongly correlated (0.72) with editing effort.
