  Neural Network based approximations of the Value function make up the core of
leading Policy Based methods such as Trust Regional Policy Optimization (TRPO)
and Proximal Policy Optimization (PPO). While this adds significant value when
dealing with very complex environments, we note that in sufficiently low State
and action space environments, a computationally expensive Neural Network
architecture offers marginal improvement over simpler Value approximation
methods. We present an implementation of Natural Actor Critic algorithms with
actor updates through Natural Policy Gradient methods. This paper proposes that
Natural Policy Gradient (NPG) methods with Linear Function Approximation as a
paradigm for value approximation may surpass the performance and speed of
Neural Network based models such as TRPO and PPO within these environments.
Over Reinforcement Learning benchmarks Cart Pole and Acrobot, we observe that
our algorithm trains much faster than complex neural network architectures, and
obtains an equivalent or greater result. This allows us to recommend the use of
NPG methods with Linear Function Approximation over TRPO and PPO for both
traditional and sparse reward low dimensional problems.
