  Efficiently utilizing rich knowledge in pretrained models has become a
critical topic in the era of large models. This work focuses on adaptively
utilizing knowledge from multiple source-pretrained models to an unlabeled
target domain without accessing the source data. Despite being a practically
useful setting, existing methods require extensive parameter tuning over each
source model, which is computationally expensive when facing abundant source
domains or larger source models. To address this challenge, we propose a novel
approach which is free of the parameter tuning over source backbones. Our
technical contribution lies in the Bi-level ATtention ENsemble (Bi-ATEN)
module, which learns both intra-domain weights and inter-domain ensemble
weights to achieve a fine balance between instance specificity and domain
consistency. By slightly tuning source bottlenecks, we achieve comparable or
even superior performance on a challenging benchmark DomainNet with less than
3% trained parameters and 8 times of throughput compared with SOTA method.
Furthermore, with minor modifications, the proposed module can be easily
equipped to existing methods and gain more than 4% performance boost. Code is
available at https://github.com/TL-UESTC/Bi-ATEN.
