  The ability to reuse previous policies is an important aspect of human
intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning
(DRL) agent needs to decide when to reuse and which source policies to reuse.
Previous methods solve this problem by introducing extra components to the
underlying algorithm, such as hierarchical high-level policies over source
policies, or estimations of source policies' value functions on the target
task. However, training these components induces either optimization
non-stationarity or heavy sampling cost, significantly impairing the
effectiveness of transfer. To tackle this problem, we propose a novel policy
reuse algorithm called Critic-gUided Policy reuse (CUP), which avoids training
any extra components and efficiently reuses source policies. CUP utilizes the
critic, a common component in actor-critic methods, to evaluate and choose
source policies. At each state, CUP chooses the source policy that has the
largest one-step improvement over the current target policy, and forms a
guidance policy. The guidance policy is theoretically guaranteed to be a
monotonic improvement over the current target policy. Then the target policy is
regularized to imitate the guidance policy to perform efficient policy search.
Empirical results demonstrate that CUP achieves efficient transfer and
significantly outperforms baseline algorithms.
