  We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that
leverages prior experience to accelerate reinforcement learning. Unlike
existing methods that learn a single skill prior from a large and diverse
dataset, our framework learns a library of different distinction skill priors
(i.e., behavior priors) from a collection of specialized datasets, and learns
how to combine them to solve a new task. This formulation allows the algorithm
to acquire a set of specialized skill priors that are more reusable for
downstream tasks; however, it also brings up additional challenges of how to
effectively combine these unstructured sets of skill priors to form a new prior
for new tasks. Specifically, it requires the agent not only to identify which
skill prior(s) to use but also how to combine them (either sequentially or
concurrently) to form a new prior. To achieve this goal, ASPiRe includes
Adaptive Weight Module (AWM) that learns to infer an adaptive weight assignment
between different skill priors and uses them to guide policy learning for
downstream tasks via weighted Kullback-Leibler divergences. Our experiments
demonstrate that ASPiRe can significantly accelerate the learning of new
downstream tasks in the presence of multiple priors and show improvement on
competitive baselines.
