  Rebuses are puzzles requiring constrained multi-step reasoning to identify a
hidden phrase from a set of images and letters. In this work, we introduce a
large collection of verbalized rebuses for the Italian language and use it to
assess the rebus-solving capabilities of state-of-the-art large language
models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
on this task, ad-hoc fine-tuning seems to improve models' performance. However,
we find that performance gains from training are largely motivated by
memorization. Our results suggest that rebus solving remains a challenging test
bed to evaluate large language models' linguistic proficiency and sequential
instruction-following skills.
