  Convolutional neural networks (CNNs) have been successfully employed in
recent years for the detection of radiological abnormalities in medical images
such as plain x-rays. To date, most studies use CNNs on individual examinations
in isolation and discard previously available clinical information. In this
study we set out to explore whether Long-Short-Term-Memory networks (LSTMs) can
be used to improve classification performance when modelling the entire
sequence of radiographs that may be available for a given patient, including
their reports. A limitation of traditional LSTMs, though, is that they
implicitly assume equally-spaced observations, whereas the radiological exams
are event-based, and therefore irregularly sampled. Using both a simulated
dataset and a large-scale chest x-ray dataset, we demonstrate that a simple
modification of the LSTM architecture, which explicitly takes into account the
time lag between consecutive observations, can boost classification
performance. Our empirical results demonstrate improved detection of commonly
reported abnormalities on chest x-rays such as cardiomegaly, consolidation,
pleural effusion and hiatus hernia.
