  Evaluating the degree of reproduction of copyright-protected content by
language models (LMs) is of significant interest to the AI and legal
communities. Although both literal and non-literal similarities are considered
by courts when assessing the degree of reproduction, prior research has focused
only on literal similarities. To bridge this gap, we introduce CopyBench, a
benchmark designed to measure both literal and non-literal copying in LM
generations. Using copyrighted fiction books as text sources, we provide
automatic evaluation protocols to assess literal and non-literal copying,
balanced against the model utility in terms of the ability to recall facts from
the copyrighted works and generate fluent completions. We find that, although
literal copying is relatively rare, two types of non-literal copying -- event
copying and character copying -- occur even in models as small as 7B
parameters. Larger models demonstrate significantly more copying, with literal
copying rates increasing from 0.2% to 10.5% and non-literal copying from 2.3%
to 6.9% when comparing Llama3-8B and 70B models, respectively. We further
evaluate the effectiveness of current strategies for mitigating copying and
show that (1) training-time alignment can reduce literal copying but may
increase non-literal copying, and (2) current inference-time mitigation methods
primarily reduce literal but not non-literal copying.
