  This paper introduces a deep neural network based method, i.e., DeepOrganNet,
to generate and visualize high-fidelity 3D / 4D organ geometric models from
single-view medical image in real time. Traditional 3D / 4D medical image
reconstruction requires near hundreds of projections, which cost insufferable
computational time and deliver undesirable high imaging / radiation dose to
human subjects. Moreover, it always needs further notorious processes to
extract the accurate 3D organ models subsequently. To our knowledge, there is
no method directly and explicitly reconstructing multiple 3D organ meshes from
a single 2D medical grayscale image on the fly. Given single-view 2D medical
images, e.g., 3D / 4D-CT projections or X-ray images, our end-to-end
DeepOrganNet framework can efficiently and effectively reconstruct 3D / 4D lung
models with a variety of geometric shapes by learning the smooth deformation
fields from multiple templates based on a trivariate tensor-product deformation
technique, leveraging an informative latent descriptor extracted from input 2D
images. The proposed method can guarantee to generate high-quality and
high-fidelity manifold meshes for 3D / 4D lung models. The major contributions
of this work are to accurately reconstruct the 3D organ shapes from 2D
single-view projection, significantly improve the procedure time to allow
on-the-fly visualization, and dramatically reduce the imaging dose for human
subjects. Experimental results are evaluated and compared with the traditional
reconstruction method and the state-of-the-art in deep learning, by using
extensive 3D and 4D examples from synthetic phantom and real patient datasets.
The proposed method only needs several milliseconds to generate organ meshes
with 10K vertices, which has a great potential to be used in real-time image
guided radiation therapy (IGRT).
