  Minimally Invasive Surgery (MIS) techniques have gained rapid popularity
among surgeons since they offer significant clinical benefits including reduced
recovery time and diminished post-operative adverse effects. However,
conventional endoscopic systems output monocular video which compromises depth
perception, spatial orientation and field of view. Suturing is one of the most
complex tasks performed under these circumstances. Key components of this tasks
are the interplay between needle holder and the surgical needle. Reliable 3D
localization of needle and instruments in real time could be used to augment
the scene with additional parameters that describe their quantitative geometric
relation, e.g. the relation between the estimated needle plane and its rotation
center and the instrument. This could contribute towards standardization and
training of basic skills and operative techniques, enhance overall surgical
performance, and reduce the risk of complications. The paper proposes an
Augmented Reality environment with quantitative and qualitative visual
representations to enhance laparoscopic training outcomes performed on a
silicone pad. This is enabled by a multi-task supervised deep neural network
which performs multi-class segmentation and depth map prediction. Scarcity of
labels has been conquered by creating a virtual environment which resembles the
surgical training scenario to generate dense depth maps and segmentation maps.
The proposed convolutional neural network was tested on real surgical training
scenarios and showed to be robust to occlusion of the needle. The network
achieves a dice score of 0.67 for surgical needle segmentation, 0.81 for needle
holder instrument segmentation and a mean absolute error of 6.5 mm for depth
estimation.
