  Ensemble of machine learning models yields improved performance as well as
robustness. However, their memory requirements and inference costs can be
prohibitively high. Knowledge distillation is an approach that allows a single
model to efficiently capture the approximate performance of an ensemble while
showing poor scalability as demand for re-training when introducing new teacher
models. In this paper, we study if we can utilize the meta-learning strategy to
directly predict the parameters of a single model with comparable performance
of an ensemble. Hereto, we introduce WeightFormer, a Transformer-based model
that can predict student network weights layer by layer in a forward pass,
according to the teacher model parameters. The proprieties of WeightFormer are
investigated on the CIFAR-10, CIFAR-100, and ImageNet datasets for model
structures of VGGNet-11, ResNet-50, and ViT-B/32, where it demonstrates that
our method can achieve approximate classification performance of an ensemble
and outperforms both the single network and standard knowledge distillation.
More encouragingly, we show that WeightFormer results can further exceeds
average ensemble with minor fine-tuning. Importantly, our task along with the
model and results can potentially lead to a new, more efficient, and scalable
paradigm of ensemble networks parameter learning.
