  Pretraining has sparked groundswell of interest in deep learning workflows to
learn from limited data and improve generalization. While this is common for 2D
image classification tasks, its application to 3D medical imaging tasks like
chest CT interpretation is limited. We explore the idea of whether pretraining
a model on realistic videos could improve performance rather than training the
model from scratch, intended for tuberculosis type classification from chest CT
scans. To incorporate both spatial and temporal features, we develop a hybrid
convolutional neural network (CNN) and recurrent neural network (RNN) model,
where the features are extracted from each axial slice of the CT scan by a CNN,
these sequence of image features are input to a RNN for classification of the
CT scan. Our model termed as ViPTT-Net, was trained on over 1300 video clips
with labels of human activities, and then fine-tuned on chest CT scans with
labels of tuberculosis type. We find that pretraining the model on videos lead
to better representations and significantly improved model validation
performance from a kappa score of 0.17 to 0.35, especially for
under-represented class samples. Our best method achieved 2nd place in the
ImageCLEF 2021 Tuberculosis - TBT classification task with a kappa score of
0.20 on the final test set with only image information (without using clinical
meta-data). All codes and models are made available.
