  Traditionally, reinforcement learning methods predict the next action based
on the current state. However, in many situations, directly applying actions to
control systems or robots is dangerous and may lead to unexpected behaviors
because action is rather low-level. In this paper, we propose a novel
hierarchical reinforcement learning framework without explicit action. Our meta
policy tries to manipulate the next optimal state and actual action is produced
by the inverse dynamics model. To stabilize the training process, we integrate
adversarial learning and information bottleneck into our framework. Under our
framework, widely available state-only demonstrations can be exploited
effectively for imitation learning. Also, prior knowledge and constraints can
be applied to meta policy. We test our algorithm in simulation tasks and its
combination with imitation learning. The experimental results show the
reliability and robustness of our algorithms.
