  Despite the recent ubiquity of large language models and their high zero-shot
prompted performance across a wide range of tasks, it is still not known how
well they perform on tasks which require processing of potentially idiomatic
language. In particular, how well do such models perform in comparison to
encoder-only models fine-tuned specifically for idiomaticity tasks? In this
work, we attempt to answer this question by looking at the performance of a
range of LLMs (both local and software-as-a-service models) on three
idiomaticity datasets: SemEval 2022 Task 2a, FLUTE, and MAGPIE. Overall, we
find that whilst these models do give competitive performance, they do not
match the results of fine-tuned task-specific models, even at the largest
scales (e.g. for GPT-4). Nevertheless, we do see consistent performance
improvements across model scale. Additionally, we investigate prompting
approaches to improve performance, and discuss the practicalities of using LLMs
for these tasks.
