  Many researchers have explored methods for hierarchical reinforcement
learning (RL) with temporal abstractions, in which abstract actions are defined
that can perform many primitive actions before terminating. However, little is
known about learning with state abstractions, in which aspects of the state
space are ignored. In previous work, we developed the MAXQ method for
hierarchical RL. In this paper, we define five conditions under which state
abstraction can be combined with the MAXQ value function decomposition. We
prove that the MAXQ-Q learning algorithm converges under these conditions and
show experimentally that state abstraction is important for the successful
application of MAXQ-Q learning.
