  This paper studies automated categorization of age-related macular
degeneration (AMD) given a multi-modal input, which consists of a color fundus
image and an optical coherence tomography (OCT) image from a specific eye.
Previous work uses a traditional method, comprised of feature extraction and
classifier training that cannot be optimized jointly. By contrast, we propose a
two-stream convolutional neural network (CNN) that is end-to-end. The CNN's
fusion layer is tailored to the need of fusing information from the fundus and
OCT streams. For generating more multi-modal training instances, we introduce
Loose Pair training, where a fundus image and an OCT image are paired based on
class labels rather than eyes. Moreover, for a visual interpretation of how the
individual modalities make contributions, we extend the class activation
mapping technique to the multi-modal scenario. Experiments on a real-world
dataset collected from an outpatient clinic justify the viability of our
proposal for multi-modal AMD categorization.
