  While deep learning has been very beneficial in data-rich settings, tasks
with smaller training set often resort to pre-training or multitask learning to
leverage data from other tasks. In this case, careful consideration is needed
to select tasks and model parameterizations such that updates from the
auxiliary tasks actually help the primary task. We seek to alleviate this
burden by formulating a model-agnostic framework that performs fine-grained
manipulation of the auxiliary task gradients. We propose to decompose auxiliary
updates into directions which help, damage or leave the primary task loss
unchanged. This allows weighting the update directions differently depending on
their impact on the problem of interest. We present a novel and efficient
algorithm for that purpose and show its advantage in practice. Our method
leverages efficient automatic differentiation procedures and randomized
singular value decomposition for scalability. We show that our framework is
generic and encompasses some prior work as particular cases. Our approach
consistently outperforms strong and widely used baselines when leveraging
out-of-distribution data for Text and Image classification tasks.
