  We propose an attention-based method that aggregates local image features to
a subject-level representation for predicting disease severity. In contrast to
classical deep learning that requires a fixed dimensional input, our method
operates on a set of image patches; hence it can accommodate variable length
input image without image resizing. The model learns a clinically interpretable
subject-level representation that is reflective of the disease severity. Our
model consists of three mutually dependent modules which regulate each other:
(1) a discriminative network that learns a fixed-length representation from
local features and maps them to disease severity; (2) an attention mechanism
that provides interpretability by focusing on the areas of the anatomy that
contribute the most to the prediction task; and (3) a generative network that
encourages the diversity of the local latent features. The generative term
ensures that the attention weights are non-degenerate while maintaining the
relevance of the local regions to the disease severity. We train our model
end-to-end in the context of a large-scale lung CT study of Chronic Obstructive
Pulmonary Disease (COPD). Our model gives state-of-the art performance in
predicting clinical measures of severity for COPD. The distribution of the
attention provides the regional relevance of lung tissue to the clinical
measurements.
