  Constructing agents with planning capabilities has long been one of the main
challenges in the pursuit of artificial intelligence. Tree-based planning
methods from AlphaGo to Muzero have enjoyed huge success in discrete domains,
such as chess and Go. Unfortunately, in real-world applications like robot
control and inverted pendulum, whose action space is normally continuous, those
tree-based planning techniques will be struggling. To address those
limitations, in this paper, we present a novel model-based reinforcement
learning frameworks called Critic PI2, which combines the benefits from
trajectory optimization, deep actor-critic learning, and model-based
reinforcement learning. Our method is evaluated for inverted pendulum models
with applicability to many continuous control systems. Extensive experiments
demonstrate that Critic PI2 achieved a new state of the art in a range of
challenging continuous domains. Furthermore, we show that planning with a
critic significantly increases the sample efficiency and real-time performance.
Our work opens a new direction toward learning the components of a model-based
planning system and how to use them.
