  Deep neural networks have demonstrated remarkable performance in medical
image analysis. However, its susceptibility to spurious correlations due to
shortcut learning raises concerns about network interpretability and
reliability. Furthermore, shortcut learning is exacerbated in medical contexts
where disease indicators are often subtle and sparse. In this paper, we propose
a novel gaze-directed Vision GNN (called GD-ViG) to leverage the visual
patterns of radiologists from gaze as expert knowledge, directing the network
toward disease-relevant regions, and thereby mitigating shortcut learning.
GD-ViG consists of a gaze map generator (GMG) and a gaze-directed classifier
(GDC). Combining the global modelling ability of GNNs with the locality of
CNNs, GMG generates the gaze map based on radiologists' visual patterns.
Notably, it eliminates the need for real gaze data during inference, enhancing
the network's practical applicability. Utilizing gaze as the expert knowledge,
the GDC directs the construction of graph structures by incorporating both
feature distances and gaze distances, enabling the network to focus on
disease-relevant foregrounds. Thereby avoiding shortcut learning and improving
the network's interpretability. The experiments on two public medical image
datasets demonstrate that GD-ViG outperforms the state-of-the-art methods, and
effectively mitigates shortcut learning. Our code is available at
https://github.com/SX-SS/GD-ViG.
