  Multi-task learning (MTL) with neural networks leverages commonalities in
tasks to improve performance, but often suffers from task interference which
reduces the benefits of transfer. To address this issue we introduce the
routing network paradigm, a novel neural network and training algorithm. A
routing network is a kind of self-organizing neural network consisting of two
components: a router and a set of one or more function blocks. A function block
may be any neural network - for example a fully-connected or a convolutional
layer. Given an input the router makes a routing decision, choosing a function
block to apply and passing the output back to the router recursively,
terminating when a fixed recursion depth is reached. In this way the routing
network dynamically composes different function blocks for each input. We
employ a collaborative multi-agent reinforcement learning (MARL) approach to
jointly train the router and function blocks. We evaluate our model against
cross-stitch networks and shared-layer baselines on multi-task settings of the
MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a
significant improvement in accuracy, with sharper convergence. In addition,
routing networks have nearly constant per-task training cost while cross-stitch
networks scale linearly with the number of tasks. On CIFAR-100 (20 tasks) we
obtain cross-stitch performance levels with an 85% reduction in training time.
