  This paper investigates the performance of the Large Language Models (LLMs)
ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the
performance, implications for didactic scenarios and assessment formats
utilizing LLMs are derived. For the analysis, 72 Python tasks for novice
programmers were selected from the free site CodingBat. Full task descriptions
were used as input to the LLMs, while the generated replies were evaluated
using CodingBat's unit tests. In addition, the general availability of textual
explanations and program code was analyzed. The results show high scores of
94.4 to 95.8% correct responses and reliable availability of textual
explanations and program code, which opens new ways to incorporate LLMs into
programming education and assessment.
