  The emergence of advanced Natural Language Processing (NLP) models like
ChatGPT has raised concerns among universities regarding AI-driven exam
completion. This paper provides a comprehensive evaluation of the proficiency
of GPT-4 and GPT-3.5 in answering a set of 42 exam papers derived from 10
distinct physics courses, administered at Durham University over the span of
2018 to 2022, totalling 593 questions and 2504 available marks. These exams,
spanning both undergraduate and postgraduate levels, include traditional
pre-COVID and adaptive COVID-era formats. Questions from the years 2018-2020
were designed for pre-COVID in person adjudicated examinations whereas the
2021-2022 exams were set for varying COVID-adapted conditions including
open-book conditions. To ensure a fair evaluation of AI performances, the exams
completed by AI were assessed by the original exam markers. However, due to
staffing constraints, only the aforementioned 593 out of the total 1280
questions were marked. GPT-4 and GPT-3.5 scored an average of 49.4\% and
38.6\%, respectively, suggesting only the weaker students would potential
improve their marks if using AI. For exams from the pre-COVID era, the average
scores for GPT-4 and GPT-3.5 were 50.8\% and 41.6\%, respectively. However,
post-COVID, these dropped to 47.5\% and 33.6\%. Thus contrary to expectations,
the change to less fact-based questions in the COVID era did not significantly
impact AI performance for the state-of-the-art models such as GPT-4. These
findings suggest that while current AI models struggle with university-level
Physics questions, an improving trend is observable. The code used for
automated AI completion is made publicly available for further research.
