  In the last years, artificial intelligence (AI) in clinical decision support
systems (CDSS) played a key role in harnessing machine learning and deep
learning architectures. Despite their promising capabilities, the lack of
transparency and explainability of AI models poses significant challenges,
particularly in medical contexts where reliability is a mandatory aspect.
Achieving transparency without compromising predictive accuracy remains a key
challenge. This paper presents a novel method, namely Rad4XCNN, to enhance the
predictive power of CNN-derived features with the interpretability inherent in
radiomic features. Rad4XCNN diverges from conventional methods based on
saliency map, by associating intelligible meaning to CNN-derived features by
means of Radiomics, offering new perspectives on explanation methods beyond
visualization maps. Using a breast cancer classification task as a case study,
we evaluated Rad4XCNN on ultrasound imaging datasets, including an online
dataset and two in-house datasets for internal and external validation. Some
key results are: i) CNN-derived features guarantee more robust accuracy when
compared against ViT-derived and radiomic features; ii) conventional
visualization map methods for explanation present several pitfalls; iii)
Rad4XCNN does not sacrifice model accuracy for their explainability; iv)
Rad4XCNN provides global explanation insights enabling the physician to analyze
the model outputs and findings. In addition, we highlight the importance of
integrating interpretability into AI models for enhanced trust and adoption in
clinical practice, emphasizing how our method can mitigate some concerns
related to explainable AI methods.
