  Reinforcement learning (RL) has been widely used in text generation to
alleviate the exposure bias issue or to utilize non-parallel datasets. The
reward function plays an important role in making RL training successful.
However, previous reward functions are typically task-specific and sparse,
restricting the use of RL. In our work, we propose a task-agnostic approach
that derives a step-wise reward function directly from a model trained with
teacher forcing. We additionally propose a simple modification to stabilize the
RL training on non-parallel datasets with our induced reward function.
Empirical results show that our method outperforms self-training and reward
regression methods on several text generation tasks, confirming the
effectiveness of our reward function.
