  In the scenario-based evaluation of machine learning models, a key problem is
how to construct test datasets that represent various scenarios. The
methodology proposed in this paper is to construct a benchmark and attach
metadata to each test case. Then a test system can be constructed with test
morphisms that filter the test cases based on metadata to form a dataset.
  The paper demonstrates this methodology with large language models for code
generation. A benchmark called ScenEval is constructed from problems in
textbooks, an online tutorial website and Stack Overflow. Filtering by scenario
is demonstrated and the test sets are used to evaluate ChatGPT for Java code
generation.
  Our experiments found that the performance of ChatGPT decreases with the
complexity of the coding task. It is weakest for advanced topics like
multi-threading, data structure algorithms and recursive methods. The Java code
generated by ChatGPT tends to be much shorter than reference solution in terms
of number of lines, while it is more likely to be more complex in both
cyclomatic and cognitive complexity metrics, if the generated code is correct.
However, the generated code is more likely to be less complex than the
reference solution if the code is incorrect.
