  COVID-19 image analysis has mostly focused on diagnostic tasks using single
timepoint scans acquired upon disease presentation or admission. We present a
deep learning-based approach to predict lung infiltrate progression from serial
chest radiographs (CXRs) of COVID-19 patients. Our method first utilizes
convolutional neural networks (CNNs) for feature extraction from patches within
the concerned lung zone, and also from neighboring and remote boundary regions.
The framework further incorporates a multi-scale Gated Recurrent Unit (GRU)
with a correlation module for effective predictions. The GRU accepts CNN
feature vectors from three different areas as input and generates a fused
representation. The correlation module attempts to minimize the correlation
loss between hidden representations of concerned and neighboring area feature
vectors, while maximizing the loss between the same from concerned and remote
regions. Further, we employ an attention module over the output hidden states
of each encoder timepoint to generate a context vector. This vector is used as
an input to a decoder module to predict patch severity grades at a future
timepoint. Finally, we ensemble the patch classification scores to calculate
patient-wise grades. Specifically, our framework predicts zone-wise disease
severity for a patient on a given day by learning representations from the
previous temporal CXRs. Our novel multi-institutional dataset comprises
sequential CXR scans from N=93 patients. Our approach outperforms transfer
learning and radiomic feature-based baseline approaches on this dataset.
