  While deep networks have achieved broad success in analyzing natural images,
when applied to medical scans, they often fail in unexcepted situations. We
investigate this challenge and focus on model sensitivity to domain shifts,
such as data sampled from different hospitals or data confounded by demographic
variables such as sex, race, etc, in the context of chest X-rays and skin
lesion images. A key finding we show empirically is that existing visual
backbones lack an appropriate prior from the architecture for reliable
generalization in these settings. Taking inspiration from medical training, we
propose giving deep networks a prior grounded in explicit medical knowledge
communicated in natural language. To this end, we introduce Knowledge-enhanced
Bottlenecks (KnoBo), a class of concept bottleneck models that incorporates
knowledge priors that constrain it to reason with clinically relevant factors
found in medical textbooks or PubMed. KnoBo uses retrieval-augmented language
models to design an appropriate concept space paired with an automatic training
procedure for recognizing the concept. We evaluate different resources of
knowledge and recognition architectures on a broad range of domain shifts
across 20 datasets. In our comprehensive evaluation with two imaging
modalities, KnoBo outperforms fine-tuned models on confounded datasets by 32.4%
on average. Finally, evaluations reveal that PubMed is a promising resource for
making medical models less sensitive to domain shift, outperforming other
resources on both diversity of information and final prediction performance.
