  Automated surface segmentation of retinal layer is important and challenging
in analyzing optical coherence tomography (OCT). Recently, many deep learning
based methods have been developed for this task and yield remarkable
performance. However, due to large spatial gap and potential mismatch between
the B-scans of OCT data, all of them are based on 2D segmentation of individual
B-scans, which may loss the continuity information across the B-scans. In
addition, 3D surface of the retina layers can provide more diagnostic
information, which is crucial in quantitative image analysis. In this study, a
novel framework based on hybrid 2D-3D convolutional neural networks (CNNs) is
proposed to obtain continuous 3D retinal layer surfaces from OCT. The 2D
features of individual B-scans are extracted by an encoder consisting of 2D
convolutions. These 2D features are then used to produce the alignment
displacement field and layer segmentation by two 3D decoders, which are coupled
via a spatial transformer module. The entire framework is trained end-to-end.
To the best of our knowledge, this is the first study that attempts 3D retinal
layer segmentation in volumetric OCT images based on CNNs. Experiments on a
publicly available dataset show that our framework achieves superior results to
state-of-the-art 2D methods in terms of both layer segmentation accuracy and
cross-B-scan 3D continuity, thus offering more clinical values than previous
works.
