  The ability to continuously acquire new knowledge and skills is crucial for
autonomous agents. Existing methods are typically based on either fixed-size
models that struggle to learn a large number of diverse behaviors, or
growing-size models that scale poorly with the number of tasks. In this work,
we aim to strike a better balance between an agent's size and performance by
designing a method that grows adaptively depending on the task sequence. We
introduce Continual Subspace of Policies (CSP), a new approach that
incrementally builds a subspace of policies for training a reinforcement
learning agent on a sequence of tasks. The subspace's high expressivity allows
CSP to perform well for many different tasks while growing sublinearly with the
number of tasks. Our method does not suffer from forgetting and displays
positive transfer to new tasks. CSP outperforms a number of popular baselines
on a wide range of scenarios from two challenging domains, Brax (locomotion)
and Continual World (manipulation).
