  Automated diagnosis prediction from medical images is a valuable resource to
support clinical decision-making. However, such systems usually need to be
trained on large amounts of annotated data, which often is scarce in the
medical domain. Zero-shot methods address this challenge by allowing a flexible
adaption to new settings with different clinical findings without relying on
labeled data. Further, to integrate automated diagnosis in the clinical
workflow, methods should be transparent and explainable, increasing medical
professionals' trust and facilitating correctness verification. In this work,
we introduce Xplainer, a novel framework for explainable zero-shot diagnosis in
the clinical setting. Xplainer adapts the classification-by-description
approach of contrastive vision-language models to the multi-label medical
diagnosis task. Specifically, instead of directly predicting a diagnosis, we
prompt the model to classify the existence of descriptive observations, which a
radiologist would look for on an X-Ray scan, and use the descriptor
probabilities to estimate the likelihood of a diagnosis. Our model is
explainable by design, as the final diagnosis prediction is directly based on
the prediction of the underlying descriptors. We evaluate Xplainer on two chest
X-ray datasets, CheXpert and ChestX-ray14, and demonstrate its effectiveness in
improving the performance and explainability of zero-shot diagnosis. Our
results suggest that Xplainer provides a more detailed understanding of the
decision-making process and can be a valuable tool for clinical diagnosis.
