  In this paper we present a novel method for learning hierarchical
representations of Markov decision processes. Our method works by partitioning
the state space into subsets, and defines subtasks for performing transitions
between the partitions. We formulate the problem of partitioning the state
space as an optimization problem that can be solved using gradient descent
given a set of sampled trajectories, making our method suitable for
high-dimensional problems with large state spaces. We empirically validate the
method, by showing that it can successfully learn a useful hierarchical
representation in a navigation domain. Once learned, the hierarchical
representation can be used to solve different tasks in the given domain, thus
generalizing knowledge across tasks.
