  Large Language Models (LLMs) zero-shot and few-shot performance are subject
to memorization and data contamination, complicating the assessment of their
validity. In literary tasks, the performance of LLMs is often correlated to the
degree of book memorization. In this work, we carry out a realistic evaluation
of LLMs for quotation attribution in novels, taking the instruction fined-tuned
version of Llama3 as an example. We design a task-specific memorization measure
and use it to show that Llama3's ability to perform quotation attribution is
positively correlated to the novel degree of memorization. However, Llama3
still performs impressively well on books it has not memorized nor seen. Data
and code will be made publicly available.
