  We consider the problem of automatically generating textual paraphrases with
modified attributes or properties, focusing on the setting without parallel
data (Hu et al., 2017; Shen et al., 2017). This setting poses challenges for
evaluation. We show that the metric of post-transfer classification accuracy is
insufficient on its own, and propose additional metrics based on semantic
preservation and fluency as well as a way to combine them into a single overall
score. We contribute new loss functions and training strategies to address the
different metrics. Semantic preservation is addressed by adding a cyclic
consistency loss and a loss based on paraphrase pairs, while fluency is
improved by integrating losses based on style-specific language models. We
experiment with a Yelp sentiment dataset and a new literature dataset that we
propose, using multiple models that extend prior work (Shen et al., 2017). We
demonstrate that our metrics correlate well with human judgments, at both the
sentence-level and system-level. Automatic and manual evaluation also show
large improvements over the baseline method of Shen et al. (2017). We hope that
our proposed metrics can speed up system development for new textual transfer
tasks while also encouraging the community to address our three complementary
aspects of transfer quality.
