  In reinforcement learning applications like robotics, agents usually need to
deal with various input/output features when specified with different
state/action spaces by their developers or physical restrictions. This
indicates unnecessary re-training from scratch and considerable sample
inefficiency, especially when agents follow similar solution steps to achieve
tasks. In this paper, we aim to transfer similar high-level goal-transition
knowledge to alleviate the challenge. Specifically, we propose PILoT, i.e.,
Planning Immediate Landmarks of Targets. PILoT utilizes the universal decoupled
policy optimization to learn a goal-conditioned state planner; then, distills a
goal-planner to plan immediate landmarks in a model-free style that can be
shared among different agents. In our experiments, we show the power of PILoT
on various transferring challenges, including few-shot transferring across
action spaces and dynamics, from low-dimensional vector states to image inputs,
from simple robot to complicated morphology; and we also illustrate a zero-shot
transfer solution from a simple 2D navigation task to the harder Ant-Maze task.
