  We explore the evolving efficacy of three generative pre-trained transformer
(GPT) models in generating answers for multiple-choice questions (MCQ) from
introductory and intermediate Python programming courses in higher education.
We focus on the differences in capabilities of the models prior to the release
of ChatGPT (Nov '22), at the time of the release, and today (i.e., Aug '23).
Recent studies have established that the abilities of the OpenAI's GPT models
to handle assessments originally designed for humans keep increasing as the
newer more capable models are released. However, the qualitative differences in
the capabilities and limitations of these models to reason about and/or analyze
programming MCQs have been under-explored. We evaluated three OpenAI's GPT
models on formative and summative MCQ assessments from three Python courses
(530 questions) focusing on the qualitative differences in the evolving
efficacy of the subsequent models. This study provides further evidence and
insight into the trajectory of the current developments where there already
exists a technology that can be utilized by students to collect passing scores,
with no effort whatsoever, on what today counts as viable programming knowledge
and skills assessments. This study could be leveraged by educators and
institutions to better understand the recent technological developments in
order to adapt the design of programming assessments as well as to fuel the
necessary discussions into how assessments in future programming classes should
be updated.
