  It has become routine to report research results where Large Language Models
(LLMs) outperform average humans in a wide range of language-related tasks, and
creative text writing is no exception. It seems natural, then, to raise the
bid: Are LLMs ready to compete in creative writing skills with a top (rather
than average) novelist? To provide an initial answer for this question, we have
carried out a contest between Patricio Pron (an awarded novelist, considered
one of the best of his generation) and GPT-4 (one of the top performing LLMs),
in the spirit of AI-human duels such as DeepBlue vs Kasparov and AlphaGo vs Lee
Sidol. We asked Pron and GPT-4 to provide thirty titles each, and then to write
short stories for both their titles and their opponent's. Then, we prepared an
evaluation rubric inspired by Boden's definition of creativity, and we
collected 5,400 manual assessments provided by literature critics and scholars.
The results of our experimentation indicate that LLMs are still far from
challenging a top human creative writer, and that reaching such level of
autonomous creative writing skills probably cannot be reached simply with
larger language models.
