  The scarcity of well-annotated medical datasets requires leveraging transfer
learning from broader datasets like ImageNet or pre-trained models like CLIP.
Model soups averages multiple fine-tuned models aiming to improve performance
on In-Domain (ID) tasks and enhance robustness against Out-of-Distribution
(OOD) datasets. However, applying these methods to the medical imaging domain
faces challenges and results in suboptimal performance. This is primarily due
to differences in error surface characteristics that stem from data
complexities such as heterogeneity, domain shift, class imbalance, and
distributional shifts between training and testing phases. To address this
issue, we propose a hierarchical merging approach that involves local and
global aggregation of models at various levels based on models' hyperparameter
configurations. Furthermore, to alleviate the need for training a large number
of models in the hyperparameter search, we introduce a computationally
efficient method using a cyclical learning rate scheduler to produce multiple
models for aggregation in the weight space. Our method demonstrates significant
improvements over the model souping approach across multiple datasets (around
6% gain in HAM10000 and CheXpert datasets) while maintaining low computational
costs for model generation and selection. Moreover, we achieve better results
on OOD datasets than model soups. The code is available at
https://github.com/BioMedIA-MBZUAI/FissionFusion.
