  We observe that incorporating a shared layer in a mixture-of-experts can lead
to performance degradation. This leads us to hypothesize that learning shared
features poses challenges in deep learning, potentially caused by the same
feature being learned as various different features. To address this issue, we
track each expert's usage frequency and merge the two most frequently selected
experts. We then update the least frequently selected expert using the
combination of experts. This approach, combined with the subsequent learning of
the router's expert selection, allows the model to determine if the most
frequently selected experts have learned the same feature differently. If they
have, the combined expert can be further trained to learn a more general
feature. Consequently, our algorithm enhances transfer learning and mitigates
catastrophic forgetting when applied to multi-domain task incremental learning.
