  Existing evaluation benchmarks of language models of code (code LMs) focus
almost exclusively on whether the LMs can generate functionally-correct code.
In real-world software engineering, developers think beyond functional
correctness. They have requirements on "how" a functionality should be
implemented to meet overall system design objectives like efficiency, security,
and maintainability. They would also trust the code LMs more if the LMs
demonstrate robust understanding of requirements and code semantics.
  We propose a new benchmark NoFunEval to evaluate code LMs on non-functional
requirements and simple classification instances for both functional and
non-functional requirements. We propose a prompting method, Coding Concepts
(CoCo), as a way for a developer to communicate the domain knowledge to the
LMs. We conduct an extensive evaluation of twenty-two code LMs. Our finding is
that they generally falter when tested on our benchmark, hinting at fundamental
blindspots in their training setups. Surprisingly, even the classification
accuracy on functional-correctness instances derived from the popular HumanEval
benchmark is low, calling in question the depth of their comprehension and the
source of their success in generating functionally-correct code in the first
place. We will release our benchmark and evaluation scripts publicly at
https://aka.ms/NoFunEval.
