  Cryptic crosswords are puzzles that rely not only on general knowledge but
also on the solver's ability to manipulate language on different levels and
deal with various types of wordplay. Previous research suggests that solving
such puzzles is a challenge even for modern NLP models. However, the abilities
of large language models (LLMs) have not yet been tested on this task. In this
paper, we establish the benchmark results for three popular LLMs -- LLaMA2,
Mistral, and ChatGPT -- showing that their performance on this task is still
far from that of humans.
