  Multi-task transfer learning based on pre-trained language encoders achieves
state-of-the-art performance across a range of tasks. Standard approaches
implicitly assume the tasks, for which we have training data, are equally
representative of the tasks we are interested in, an assumption which is often
hard to justify. This paper presents a more agnostic approach to multi-task
transfer learning, which uses automated curriculum learning to minimize a new
family of worst-case-aware losses across tasks. Not only do these losses lead
to better performance on outlier tasks; they also lead to better performance in
zero-shot and few-shot transfer settings.
