  The goal of reinforcement learning (RL) is to let an agent learn an optimal
control policy in an unknown environment so that future expected rewards are
maximized. The model-free RL approach directly learns the policy based on data
samples. Although using many samples tends to improve the accuracy of policy
learning, collecting a large number of samples is often expensive in practice.
On the other hand, the model-based RL approach first estimates the transition
model of the environment and then learns the policy based on the estimated
transition model. Thus, if the transition model is accurately learned from a
small amount of data, the model-based approach can perform better than the
model-free approach. In this paper, we propose a novel model-based RL method by
combining a recently proposed model-free policy search method called policy
gradients with parameter-based exploration and the state-of-the-art transition
model estimator called least-squares conditional density estimation. Through
experiments, we demonstrate the practical usefulness of the proposed method.
