  Formulating expert policies as macro actions promises to alleviate the
long-horizon issue via structured exploration and efficient credit assignment.
However, traditional option-based multi-policy transfer methods suffer from
inefficient exploration of macro action's length and insufficient exploitation
of useful long-duration macro actions. In this paper, a novel algorithm named
EASpace (Enhanced Action Space) is proposed, which formulates macro actions in
an alternative form to accelerate the learning process using multiple available
sub-optimal expert policies. Specifically, EASpace formulates each expert
policy into multiple macro actions with different execution {times}. All the
macro actions are then integrated into the primitive action space directly. An
intrinsic reward, which is proportional to the execution time of macro actions,
is introduced to encourage the exploitation of useful macro actions. The
corresponding learning rule that is similar to Intra-option Q-learning is
employed to improve the data efficiency. Theoretical analysis is presented to
show the convergence of the proposed learning rule. The efficiency of EASpace
is illustrated by a grid-based game and a multi-agent pursuit problem. The
proposed algorithm is also implemented in physical systems to validate its
effectiveness.
