  In deep multi-task learning, weights of task-specific networks are shared
between tasks to improve performance on each single one. Since the question,
which weights to share between layers, is difficult to answer, human-designed
architectures often share everything but a last task-specific layer. In many
cases, this simplistic approach severely limits performance. Instead, we
propose an algorithm to learn the assignment between a shared set of weights
and task-specific layers. To optimize the non-differentiable assignment and at
the same time train the differentiable weights, learning takes place via a
combination of natural evolution strategy and stochastic gradient descent. The
end result are task-specific networks that share weights but allow independent
inference. They achieve lower test errors than baselines and methods from
literature on three multi-task learning datasets.
