  Language models (LMs) derive their capabilities from extensive training on
diverse data, including potentially copyrighted material. These models can
memorize and generate content similar to their training data, posing potential
concerns. Therefore, model creators are motivated to develop mitigation methods
that prevent generating protected content. We term this procedure as copyright
takedowns for LMs, noting the conceptual similarity to (but legal distinction
from) the DMCA takedown This paper introduces the first evaluation of the
feasibility and side effects of copyright takedowns for LMs. We propose
CoTaEval, an evaluation framework to assess the effectiveness of copyright
takedown methods, the impact on the model's ability to retain uncopyrightable
factual knowledge from the training data whose recitation is embargoed, and how
well the model maintains its general utility and efficiency. We examine several
strategies, including adding system prompts, decoding-time filtering
interventions, and unlearning approaches. Our findings indicate that no tested
method excels across all metrics, showing significant room for research in this
unique problem setting and indicating potential unresolved challenges for live
policy proposals.
