  Pre-trained multimodal foundation models have demonstrated remarkable
generalizability but pose challenges for deployment due to their large sizes.
One effective approach to reducing their sizes is layerwise distillation,
wherein small student models are trained to match the hidden representations of
large teacher models at each layer. Motivated by our observation that certain
architecture components, referred to as modules, contribute more significantly
to the student's performance than others, we propose to track the contributions
of individual modules by recording the loss decrement after distillation each
module and choose the module with a greater contribution to distill more
frequently. Such an approach can be naturally formulated as a multi-armed
bandit (MAB) problem, where modules and loss decrements are considered as arms
and rewards, respectively. We then develop a modified-Thompson sampling
algorithm named OPTIMA to address the nonstationarity of module contributions
resulting from model updating. Specifically, we leverage the observed
contributions in recent history to estimate the changing contribution of each
module and select modules based on these estimations to maximize the cumulative
contribution. We evaluate the effectiveness of OPTIMA through distillation
experiments on various multimodal understanding and image captioning tasks,
using the CoCa-Large model (Yu et al., 2022) as the teacher model.
