  Generating responses in a targeted style is a useful yet challenging task,
especially in the absence of parallel data. With limited data, existing methods
tend to generate responses that are either less stylized or less
context-relevant. We propose StyleFusion, which bridges conversation modeling
and non-parallel style transfer by sharing a structured latent space. This
structure allows the system to generate stylized relevant responses by sampling
in the neighborhood of the conversation model prediction, and continuously
control the style level. We demonstrate this method using dialogues from Reddit
data and two sets of sentences with distinct styles (arXiv and Sherlock Holmes
novels). Automatic and human evaluation show that, without sacrificing
appropriateness, the system generates responses of the targeted style and
outperforms competitive baselines.
