  In the burgeoning field of artificial intelligence (AI), understanding the
capabilities and limitations of programming-oriented models is crucial. This
paper presents a novel evaluation of the programming proficiency of Generative
Pretrained Transformer (GPT) models, specifically GPT-3.5 and GPT-4, against
coding problems of varying difficulty levels drawn from Codewars. The
experiments reveal a distinct boundary at the 3kyu level, beyond which these
GPT models struggle to provide solutions. These findings led to the proposal of
a measure for coding problem complexity that incorporates both problem
difficulty and the time required for solution. The research emphasizes the need
for validation and creative thinking capabilities in AI models to better
emulate human problem-solving techniques. Future work aims to refine this
proposed complexity measure, enhance AI models with these suggested
capabilities, and develop an objective measure for programming problem
difficulty. The results of this research offer invaluable insights for
improving AI programming capabilities and advancing the frontier of AI
problem-solving abilities.
