  In the growing domain of scientific machine learning, in-context operator
learning has shown notable potential in building foundation models, as in this
framework the model is trained to learn operators and solve differential
equations using prompted data, during the inference stage without weight
updates. However, the current model's overdependence on function data overlooks
the invaluable human insight into the operator. To address this, we present a
transformation of in-context operator learning into a multi-modal paradigm. In
particular, we take inspiration from the recent success of large language
models, and propose using "captions" to integrate human knowledge about the
operator, expressed through natural language descriptions and equations. Also,
we introduce a novel approach to train a language-model-like architecture, or
directly fine-tune existing language models, for in-context operator learning.
We beat the baseline on single-modal learning tasks, and also demonstrated the
effectiveness of multi-modal learning in enhancing performance and reducing
function data requirements. The proposed method not only significantly enhanced
the development of the in-context operator learning paradigm, but also created
a new path for the application of language models.
