  Pathological structures in medical images are typically deviations from the
expected anatomy of a patient. While clinicians consider this interplay between
anatomy and pathology, recent deep learning algorithms specialize in
recognizing either one of the two, rarely considering the patient's body from
such a joint perspective. In this paper, we develop a generalist segmentation
model that combines anatomical and pathological information, aiming to enhance
the segmentation accuracy of pathological features. Our Anatomy-Pathology
Exchange (APEx) training utilizes a query-based segmentation transformer which
decodes a joint feature space into query-representations for human anatomy and
interleaves them via a mixing strategy into the pathology-decoder for
anatomy-informed pathology predictions. In doing so, we are able to report the
best results across the board on FDG-PET-CT and Chest X-Ray pathology
segmentation tasks with a margin of up to 3.3% as compared to strong baseline
methods. Code and models will be publicly available at
github.com/alexanderjaus/APEx.
