  Memorization in large language models (LLMs) is a growing concern. LLMs have
been shown to easily reproduce parts of their training data, including
copyrighted work. This is an important problem to solve, as it may violate
existing copyright laws as well as the European AI Act. In this work, we
propose a systematic analysis to quantify the extent of potential copyright
infringements in LLMs using European law as an example. Unlike previous work,
we evaluate instruction-finetuned models in a realistic end-user scenario. Our
analysis builds on a proposed threshold of 160 characters, which we borrow from
the German Copyright Service Provider Act and a fuzzy text matching algorithm
to identify potentially copyright-infringing textual reproductions. The
specificity of countermeasures against copyright infringement is analyzed by
comparing model behavior on copyrighted and public domain data. We investigate
what behaviors models show instead of producing protected text (such as refusal
or hallucination) and provide a first legal assessment of these behaviors. We
find that there are huge differences in copyright compliance, specificity, and
appropriate refusal among popular LLMs. Alpaca, GPT 4, GPT 3.5, and Luminous
perform best in our comparison, with OpenGPT-X, Alpaca, and Luminous producing
a particularly low absolute number of potential copyright violations. Code will
be published soon.
