  In the paraphrase generation task, source sentences often contain phrases
that should not be altered. Which phrases, however, can be context dependent
and can vary by application. Our solution to this challenge is to provide the
user with explicit tags that can be placed around any arbitrary segment of text
to mean "don't change me!" when generating a paraphrase; the model learns to
explicitly copy these phrases to the output. The contribution of this work is a
novel data generation technique using distant supervision that allows us to
start with a pretrained sequence-to-sequence model and fine-tune a paraphrase
generator that exhibits this behavior, allowing user-controllable paraphrase
generation. Additionally, we modify the loss during fine-tuning to explicitly
encourage diversity in model output. Our technique is language agnostic, and we
report experiments in English and Chinese.
