  Deep neural networks have demonstrated very promising performance on accurate
segmentation of challenging organs (e.g., pancreas) in abdominal CT and MRI
scans. The current deep learning approaches conduct pancreas segmentation by
processing sequences of 2D image slices independently through deep, dense
per-pixel masking for each image, without explicitly enforcing spatial
consistency constraint on segmentation of successive slices. We propose a new
convolutional/recurrent neural network architecture to address the contextual
learning and segmentation consistency problem. A deep convolutional sub-network
is first designed and pre-trained from scratch. The output layer of this
network module is then connected to recurrent layers and can be fine-tuned for
contextual learning, in an end-to-end manner. Our recurrent sub-network is a
type of Long short-term memory (LSTM) network that performs segmentation on an
image by integrating its neighboring slice segmentation predictions, in the
form of a dependent sequence processing. Additionally, a novel
segmentation-direct loss function (named Jaccard Loss) is proposed and deep
networks are trained to optimize Jaccard Index (JI) directly. Extensive
experiments are conducted to validate our proposed deep models, on quantitative
pancreas segmentation using both CT and MRI scans. Our method outperforms the
state-of-the-art work on CT [11] and MRI pancreas segmentation [1],
respectively.
