  This paper proposes a step toward obtaining general models of knowledge for
facial analysis, by addressing the question of multi-source transfer learning.
More precisely, the proposed approach consists in two successive training
steps: the first one consists in applying a combination operator to define a
common embedding for the multiple sources materialized by different existing
trained models. The proposed operator relies on an auto-encoder, trained on a
large dataset, efficient both in terms of compression ratio and transfer
learning performance. In a second step we exploit a distillation approach to
obtain a lightweight student model mimicking the collection of the fused
existing models. This model outperforms its teacher on novel tasks, achieving
results on par with state-of-the-art methods on 15 facial analysis tasks (and
domains), at an affordable training cost. Moreover, this student has 75 times
less parameters than the original teacher and can be applied to a variety of
novel face-related tasks.
