  Previous work has shown the potential of deep learning to predict renal
obstruction using kidney ultrasound images. However, these image-based
classifiers have been trained with the goal of single-visit inference in mind.
We compare methods from video action recognition (i.e. convolutional pooling,
LSTM, TSM) to adapt single-visit convolutional models to handle multiple visit
inference. We demonstrate that incorporating images from a patient's past
hospital visits provides only a small benefit for the prediction of obstructive
hydronephrosis. Therefore, inclusion of prior ultrasounds is beneficial, but
prediction based on the latest ultrasound is sufficient for patient risk
stratification.
