  The Mixture-of-Experts (MoE) has gained increasing attention in studying
Large Vision-Language Models (LVLMs). It uses a sparse model to replace the
dense model, achieving comparable performance while activating fewer parameters
during inference, thus significantly reducing the inference cost. Existing MoE
methods in LVLMs encourage different experts to handle different tokens, and
they usually employ a router to predict the routing of each token. However, the
predictions are based solely on sample features and do not truly reveal the
optimization directions of tokens. This may lead to severe optimization
interference between different tokens assigned to an expert. To address this
problem, this paper proposes a novel method based on token-level gradient
analysis, i.e., Solving Token Gradient Conflict (STGC). Specifically, we first
use token-level gradients to identify conflicting tokens in experts. After
that, we add a specialized loss tailored to eliminate conflicts among tokens
within each expert. Our method can serve as a plug-in for diverse Large
Vision-Language Models, and extensive experimental results demonstrate its
effectiveness. The code will be publicly available at
https://github.com/longrongyang/STGC.
