  In the field of Medical Imaging, extensive research has been dedicated to
leveraging its potential in uncovering critical diagnostic features in
patients. Artificial Intelligence (AI)-driven medical diagnosis relies on
sophisticated machine learning and deep learning models to analyze, detect, and
identify diseases from medical images. Despite the remarkable performance of
these models, characterized by high accuracy, they grapple with trustworthiness
issues. The introduction of a subtle perturbation to the original image
empowers adversaries to manipulate the prediction output, redirecting it to
other targeted or untargeted classes. Furthermore, the scarcity of publicly
available medical images, constituting a bottleneck for reliable training, has
led contemporary algorithms to depend on pretrained models grounded on a large
set of natural images -- a practice referred to as transfer learning. However,
a significant {\em domain discrepancy} exists between natural and medical
images, which causes AI models resulting from transfer learning to exhibit
heightened {\em vulnerability} to adversarial attacks. This paper proposes a
{\em domain assimilation} approach that introduces texture and color adaptation
into transfer learning, followed by a texture preservation component to
suppress undesired distortion. We systematically analyze the performance of
transfer learning in the face of various adversarial attacks under different
data modalities, with the overarching goal of fortifying the model's robustness
and security in medical imaging tasks. The results demonstrate high
effectiveness in reducing attack efficacy, contributing toward more trustworthy
transfer learning in biomedical applications.
