  Large language models (LLMs) can now generate and recognize text in a wide
range of styles and genres, including highly specialized, creative genres like
poetry. But what do LLMs really know about poetry? What can they know about
poetry? We develop a task to evaluate how well LLMs recognize a specific aspect
of poetry, poetic form, for more than 20 forms and formal elements in the
English language. Poetic form captures many different poetic features,
including rhyme scheme, meter, and word or line repetition. We use this task to
reflect on LLMs' current poetic capabilities, as well as the challenges and
pitfalls of creating NLP benchmarks for poetry and for other creative tasks. In
particular, we use this task to audit and reflect on the poems included in
popular pretraining datasets. Our findings have implications for NLP
researchers interested in model evaluation, digital humanities and cultural
analytics scholars, and cultural heritage professionals.
