  In the context of medical artificial intelligence, this study explores the
vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a
Vision Language Foundation model, under targeted attacks. Leveraging the Kather
Colon dataset with 7,180 H&E images across nine tissue types, our investigation
employs Projected Gradient Descent (PGD) adversarial perturbation attacks to
induce misclassifications intentionally. The outcomes reveal a 100% success
rate in manipulating PLIP's predictions, underscoring its susceptibility to
adversarial perturbations. The qualitative analysis of adversarial examples
delves into the interpretability challenges, shedding light on nuanced changes
in predictions induced by adversarial manipulations. These findings contribute
crucial insights into the interpretability, domain adaptation, and
trustworthiness of Vision Language Models in medical imaging. The study
emphasizes the pressing need for robust defenses to ensure the reliability of
AI models. The source codes for this experiment can be found at
https://github.com/jaiprakash1824/VLM_Adv_Attack.
