  In this paper, we consider the problem of disease diagnosis. Unlike the
conventional learning paradigm that treats labels independently, we propose a
knowledge-enhanced framework, that enables training visual representation with
the guidance of medical domain knowledge. In particular, we make the following
contributions: First, to explicitly incorporate experts' knowledge, we propose
to learn a neural representation for the medical knowledge graph via
contrastive learning, implicitly establishing relations between different
medical concepts. Second, while training the visual encoder, we keep the
parameters of the knowledge encoder frozen and propose to learn a set of prompt
vectors for efficient adaptation. Third, we adopt a Transformer-based
disease-query module for cross-model fusion, which naturally enables
explainable diagnosis results via cross attention. To validate the
effectiveness of our proposed framework, we conduct thorough experiments on
three x-ray imaging datasets across different anatomy structures, showing our
model is able to exploit the implicit relations between diseases/findings, thus
is beneficial to the commonly encountered problem in the medical domain,
namely, long-tailed and zero-shot recognition, which conventional methods
either struggle or completely fail to realize.
