  Multi-source transfer learning is an effective solution to data scarcity by
utilizing multiple source tasks for the learning of the target task. However,
access to source data and model details is limited in the era of commercial
models, giving rise to the setting of multi-source-free (MSF) transfer learning
that aims to leverage source domain knowledge without such access. As a newly
defined problem paradigm, MSF transfer learning remains largely underexplored
and not clearly formulated. In this work, we adopt an information theoretic
perspective on it and propose a framework named H-ensemble, which dynamically
learns the optimal linear combination, or ensemble, of source models for the
target task, using a generalization of maximal correlation regression. The
ensemble weights are optimized by maximizing an information theoretic metric
for transferability. Compared to previous works, H-ensemble is characterized
by: 1) its adaptability to a novel and realistic MSF setting for few-shot
target tasks, 2) theoretical reliability, 3) a lightweight structure easy to
interpret and adapt. Our method is empirically validated by ablation studies,
along with extensive comparative analysis with other task ensemble and transfer
learning methods. We show that the H-ensemble can successfully learn the
optimal task ensemble, as well as outperform prior arts.
