  Fine-tuning pre-trained Vision-Language Models (VLMs) has shown remarkable
capabilities in medical image and textual depiction synergy. Nevertheless, many
pre-training datasets are restricted by patient privacy concerns, potentially
containing noise that can adversely affect downstream performance. Moreover,
the growing reliance on multi-modal generation exacerbates this issue because
of its susceptibility to adversarial attacks. To investigate how VLMs trained
on adversarial noisy data perform on downstream medical tasks, we first craft
noisy upstream datasets using multi-modal adversarial attacks. Through our
comprehensive analysis, we unveil that moderate noise enhances model robustness
and transferability, but increasing noise levels negatively impact downstream
task performance. To mitigate this issue, we propose rectify adversarial noise
(RAN) framework, a recipe designed to effectively defend adversarial attacks
and rectify the influence of upstream noise during fine-tuning.
