  Local policy search is performed by most Deep Reinforcement Learning (D-RL)
methods, which increases the risk of getting trapped in a local minimum.
Furthermore, the availability of a simulation model is not fully exploited in
D-RL even in simulation-based training, which potentially decreases efficiency.
To better exploit simulation models in policy search, we propose to integrate a
kinodynamic planner in the exploration strategy and to learn a control policy
in an offline fashion from the generated environment interactions. We call the
resulting model-based reinforcement learning method PPS (Planning for Policy
Search). We compare PPS with state-of-the-art D-RL methods in typical RL
settings including underactuated systems. The comparison shows that PPS, guided
by the kinodynamic planner, collects data from a wider region of the state
space. This generates training data that helps PPS discover better policies.
