  Recovering the latent factors of variation of high dimensional data has so
far focused on simple synthetic settings. Mostly building on unsupervised and
weakly-supervised objectives, prior work missed out on the positive
implications for representation learning on real world data. In this work, we
propose to leverage knowledge extracted from a diversified set of supervised
tasks to learn a common disentangled representation. Assuming each supervised
task only depends on an unknown subset of the factors of variation, we
disentangle the feature space of a supervised multi-task model, with features
activating sparsely across different tasks and information being shared as
appropriate. Importantly, we never directly observe the factors of variations
but establish that access to multiple tasks is sufficient for identifiability
under sufficiency and minimality assumptions. We validate our approach on six
real world distribution shift benchmarks, and different data modalities
(images, text), demonstrating how disentangled representations can be
transferred to real settings.
