  Multi-task learning holds the promise of less data, parameters, and time than
training of separate models. We propose a method to automatically search over
multi-task architectures while taking resource constraints into consideration.
We propose a search space that compactly represents different parameter sharing
strategies. This provides more effective coverage and sampling of the space of
multi-task architectures. We also present a method for quick evaluation of
different architectures by using feature distillation. Together these
contributions allow us to quickly optimize for efficient multi-task models. We
benchmark on Visual Decathlon, demonstrating that we can automatically search
for and identify multi-task architectures that effectively make trade-offs
between task resource requirements while achieving a high level of final
performance.
