  In this paper, we present a large-scale evaluation probing GPT-4V's
capabilities and limitations for biomedical image analysis. GPT-4V represents a
breakthrough in artificial general intelligence (AGI) for computer vision, with
applications in the biomedical domain. We assess GPT-4V's performance across 16
medical imaging categories, including radiology, oncology, ophthalmology,
pathology, and more. Tasks include modality recognition, anatomy localization,
disease diagnosis, report generation, and lesion detection. The extensive
experiments provide insights into GPT-4V's strengths and weaknesses. Results
show GPT-4V's proficiency in modality and anatomy recognition but difficulty
with disease diagnosis and localization. GPT-4V excels at diagnostic report
generation, indicating strong image captioning skills. While promising for
biomedical imaging AI, GPT-4V requires further enhancement and validation
before clinical deployment. We emphasize responsible development and testing
for trustworthy integration of biomedical AGI. This rigorous evaluation of
GPT-4V on diverse medical images advances understanding of multimodal large
language models (LLMs) and guides future work toward impactful healthcare
applications.
