  Multi-task reinforcement learning endeavors to accomplish a set of different
tasks with a single policy. To enhance data efficiency by sharing parameters
across multiple tasks, a common practice segments the network into distinct
modules and trains a routing network to recombine these modules into
task-specific policies. However, existing routing approaches employ a fixed
number of modules for all tasks, neglecting that tasks with varying
difficulties commonly require varying amounts of knowledge. This work presents
a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of
certain intermediate modules, thereby flexibly choosing different numbers of
modules for each task. Under this framework, we further introduce a ResRouting
method to address the issue of disparate routing paths between behavior and
target policies during off-policy training. In addition, we design an automatic
route-balancing mechanism to encourage continued routing exploration for
unmastered tasks without disturbing the routing of mastered ones. We conduct
extensive experiments on various robotics manipulation tasks in the Meta-World
benchmark, where D2R achieves state-of-the-art performance with significantly
improved learning efficiency.
