  Foundation models, often pre-trained with large-scale data, have achieved
paramount success in jump-starting various vision and language applications.
Recent advances further enable adapting foundation models in downstream tasks
efficiently using only a few training samples, e.g., in-context learning. Yet,
the application of such learning paradigms in medical image analysis remains
scarce due to the shortage of publicly accessible data and benchmarks. In this
paper, we aim at approaches adapting the foundation models for medical image
classification and present a novel dataset and benchmark for the evaluation,
i.e., examining the overall performance of accommodating the large-scale
foundation models downstream on a set of diverse real-world clinical tasks. We
collect five sets of medical imaging data from multiple institutes targeting a
variety of real-world clinical tasks (22,349 images in total), i.e., thoracic
diseases screening in X-rays, pathological lesion tissue screening, lesion
detection in endoscopy images, neonatal jaundice evaluation, and diabetic
retinopathy grading. Results of multiple baseline methods are demonstrated
using the proposed dataset from both accuracy and cost-effective perspectives.
