  We consider the problem of autonomously learning reusable temporally extended
actions, or options, in reinforcement learning. While options can speed up
transfer learning by serving as reusable building blocks, learning reusable
options for unknown task distribution remains challenging. Motivated by the
recent success of mutual information (MI) based skill learning, we hypothesize
that more diverse options are more reusable. To this end, we propose a method
for learning termination conditions of options by maximizing MI between options
and corresponding state transitions. We derive a scalable approximation of this
MI maximization via gradient ascent, yielding the InfoMax Termination Critic
(IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves
the diversity of learned options without extrinsic rewards combined with an
intrinsic option learning method. Moreover, we test the reusability of learned
options by transferring options into various tasks, confirming that IMTC helps
quick adaptation, especially in complex domains where an agent needs to
manipulate objects.
