  It has been suggested that large language models such as GPT-4 have acquired
some form of understanding beyond the correlations among the words in text
including some understanding of mathematics as well. Here, we perform a
critical inquiry into this claim by evaluating the mathematical understanding
of the GPT-4 model. Considering that GPT-4's training set is a secret, it is
not straightforward to evaluate whether the model's correct answers are based
on a mathematical understanding or based on replication of proofs that the
model has seen before. We specifically craft mathematical questions which their
formal proofs are not readily available on the web, proofs that are more likely
not seen by the GPT-4. We see that GPT-4 is unable to solve those problems
despite their simplicity. It is hard to find scientific evidence suggesting
that GPT-4 has acquired an understanding of even basic mathematical concepts. A
straightforward way to find failure modes of GPT-4 in theorem proving is to
craft questions where their formal proofs are not available on the web. Our
finding suggests that GPT-4's ability is to reproduce, rephrase, and polish the
mathematical proofs that it has seen before, and not in grasping mathematical
concepts. We also see that GPT-4's ability to prove mathematical theorems is
continuously expanding over time despite the claim that it is a fixed model. We
suggest that the task of proving mathematical theorems in formal language is
comparable to the methods used in search engines such as Google while
predicting the next word in a sentence may be a misguided approach, a recipe
that often leads to excessive extrapolation and eventual failures. Prompting
the GPT-4 over and over may benefit the GPT-4 and the OpenAI, but we question
whether it is valuable for machine learning or for theorem proving.
