  This paper develops an inverse reinforcement learning algorithm aimed at
recovering a reward function from the observed actions of an agent. We
introduce a strategy to flexibly handle different types of actions with two
approximations of the Bellman Optimality Equation, and a Bellman Gradient
Iteration method to compute the gradient of the Q-value with respect to the
reward function. These methods allow us to build a differentiable relation
between the Q-value and the reward function and learn an approximately optimal
reward function with gradient methods. We test the proposed method in two
simulated environments by evaluating the accuracy of different approximations
and comparing the proposed method with existing solutions. The results show
that even with a linear reward function, the proposed method has a comparable
accuracy with the state-of-the-art method adopting a non-linear reward
function, and the proposed method is more flexible because it is defined on
observed actions instead of trajectories.
