  Reinforcement learning (RL) algorithms have been successfully applied to a
range of challenging sequential decision making and control tasks. In this
paper, we classify RL into direct and indirect RL according to how they seek
the optimal policy of the Markov decision process problem. The former solves
the optimal policy by directly maximizing an objective function using gradient
descent methods, in which the objective function is usually the expectation of
accumulative future rewards. The latter indirectly finds the optimal policy by
solving the Bellman equation, which is the sufficient and necessary condition
from Bellman's principle of optimality. We study policy gradient forms of
direct and indirect RL and show that both of them can derive the actor-critic
architecture and can be unified into a policy gradient with the approximate
value function and the stationary state distribution, revealing the equivalence
of direct and indirect RL. We employ a Gridworld task to verify the influence
of different forms of policy gradient, suggesting their differences and
relationships experimentally. Finally, we classify current mainstream RL
algorithms using the direct and indirect taxonomy, together with other ones
including value-based and policy-based, model-based and model-free.
