  Real-world data often exhibit extreme imbalances and out-of-distribution
(OOD) instances, which significantly biases the model training. While it has
been extensively studied in vision and language domains separately, the impact
of long-tailed open worlds on multi-modal large language models (MLLMs) has
been largely overlooked. In this paper, we first demonstrate the susceptibility
and vulnerability of vision-language models to significant biases caused by
tail drift and out-of-distribution (OOD) drift during both the pre-training and
fine-tuning stages. To eliminate the bias from different sources, we integrate
the tailed drift adaptation and OOD drift detection into a unified framework by
extending the concept drift theory to multi-modal. Specifically, a
T-distribution-based drift adapter is proposed to effectively mitigate the bias
induced by the long-tailed problem, which also facilitates the model in
distinguishing OOD data through explicit distribution modelling. Extensive
experiments show significant improvements in our model's ability to adapt to
tailed drift and OOD drift. Moreover, it enhances the efficiency and accuracy
of image-text alignment in vision language model pre-training, particularly in
the long-tail open world scenario. Furthermore, we create a set of multi-modal
datasets called OpenMMlo, specifically tailored for the long-tailed open world
scenario, to validate our findings. To foster the development of the
multi-modal community, we have made both OpenMMlo datasets and our code
publicly available at: https://github.com/Anonymous0Knight/ConceptDriftMLLMs.
