  Approximate dynamic programming algorithms, such as approximate value
iteration, have been successfully applied to many complex reinforcement
learning tasks, and a better approximate dynamic programming algorithm is
expected to further extend the applicability of reinforcement learning to
various tasks. In this paper we propose a new, robust dynamic programming
algorithm that unifies value iteration, advantage learning, and dynamic policy
programming. We call it generalized value iteration (GVI) and its approximated
version, approximate GVI (AGVI). We show AGVI's performance guarantee, which
includes performance guarantees for existing algorithms, as special cases. We
discuss theoretical weaknesses of existing algorithms, and explain the
advantages of AGVI. Numerical experiments in a simple environment support
theoretical arguments, and suggest that AGVI is a promising alternative to
previous algorithms.
