  In this working notes paper, we describe IBM Research AI (Almaden) team's
participation in the ImageCLEF 2019 VQA-Med competition. The challenge consists
of four question-answering tasks based on radiology images. The diversity of
imaging modalities, organs and disease types combined with a small imbalanced
training set made this a highly complex problem. To overcome these
difficulties, we implemented a modular pipeline architecture that utilized
transfer learning and multi-task learning. Our findings led to the development
of a novel model called Supporting Facts Network (SFN). The main idea behind
SFN is to cross-utilize information from upstream tasks to improve the accuracy
on harder downstream ones. This approach significantly improved the scores
achieved in the validation set (18 point improvement in F-1 score). Finally, we
submitted four runs to the competition and were ranked seventh.
