  In Multi-Task Learning (MTL), tasks may compete and limit the performance
achieved on each other, rather than guiding the optimization to a solution,
superior to all its single-task trained counterparts. Since there is often not
a unique solution optimal for all tasks, practitioners have to balance
tradeoffs between tasks' performance, and resort to optimality in the Pareto
sense. Most MTL methodologies either completely neglect this aspect, and
instead of aiming at learning a Pareto Front, produce one solution predefined
by their optimization schemes, or produce diverse but discrete solutions.
Recent approaches parameterize the Pareto Front via neural networks, leading to
complex mappings from tradeoff to objective space. In this paper, we conjecture
that the Pareto Front admits a linear parameterization in parameter space,
which leads us to propose \textit{Pareto Manifold Learning}, an ensembling
method in weight space. Our approach produces a continuous Pareto Front in a
single training run, that allows to modulate the performance on each task
during inference. Experiments on multi-task learning benchmarks, ranging from
image classification to tabular datasets and scene understanding, show that
\textit{Pareto Manifold Learning} outperforms state-of-the-art single-point
algorithms, while learning a better Pareto parameterization than multi-point
baselines.
