  Watermarking of language model outputs enables statistical detection of
model-generated text, which can mitigate harms and misuses of language models.
Existing watermarking strategies operate by altering the decoder of an existing
language model. In this paper, we ask whether language models can directly
learn to generate watermarked text, which would have significant implications
for the real-world deployment of watermarks. First, learned watermarks could be
used to build open models that naturally generate watermarked text, enabling
watermarking for open models, where users can control the decoding procedure.
Second, if watermarking is used to determine the provenance of generated text,
an adversary can hurt the reputation of a victim model by spoofing its
watermark and generating damaging watermarked text. To investigate the
learnability of watermarks, we propose watermark distillation, which trains a
student model to behave like a teacher model that uses decoding-based
watermarking. We test our approach on three decoding-based watermarking
strategies and various hyperparameter settings, finding that models can learn
to generate watermarked text with high detectability. We also find limitations
to learnability, including the loss of watermarking capabilities under
fine-tuning on normal text and high sample complexity when learning
low-distortion watermarks.
