  Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus,
focus on the code generation tasks. Namely, they contain a natural language
description of a problem and ask the LLM to write code to solve the problem. We
argue that they do not capture all capabilities needed to assess the quality of
a code LLM. In this paper, we propose a code LLM maturity model, based on the
postcondition generation problem, to access a more complete set of code LLM
capabilities. We choose the postcondition generation problem as it requires the
code LLM to understand the code including semantics, natural language, and also
have the capability to generate unambiguous postconditions in programming
languages (i.e., the generation capablity). Moreover, postconditions have
various types, requiring different levels of these capabilities, making it
suitable to evaluate the maturity of the code LLM. Based on our designed
maturity model, we augment the EvalPlus dataset to a postcondition testing
benchmark, and evaluated several open-sourced models. Our results highlight the
necessary improvements needed for better LLMs for code. Code:
https://github.com/MatureModel/PostcondGen
