  Inevitable domain and task discrepancies in real-world scenarios can impair
the generalization performance of the pre-trained deep models for medical data.
Therefore, we audaciously propose that we should build a general-purpose
medical AI system that can be seamlessly adapted to downstream domains/tasks.
Since the domain/task adaption procedures usually involve additional labeling
work for the target data, designing a data-efficient adaption algorithm is
desired to save the cost of transferring the learned knowledge. Our recent work
found that vision-language models (VLMs) are efficient learners with
extraordinary cross-domain ability. Therefore, in this work, we further explore
the possibility of leveraging pre-trained VLMs as medical foundation models for
building general-purpose medical AI, where we thoroughly investigate three
machine-learning paradigms, i.e., domain/task-specialized learning, joint
learning, and continual learning, for training the VLMs and evaluate their
generalization performance on cross-domain and cross-task test sets. To
alleviate the catastrophic forgetting during sequential training, we employ
rehearsal learning and receive a sharp boost in terms of generalization
capability. In a nutshell, our empirical evidence suggests that continual
learning may be a practical and efficient learning paradigm for the medical
foundation model. And we hope researchers can use our empirical evidence as
basement to further explore the path toward medical foundation model.
