  Directing of the task-specific attention while tracking instrument in surgery
holds great potential in robot-assisted intervention. For this purpose, we
propose an end-to-end trainable multitask learning (MTL) model for real-time
surgical instrument segmentation and attention prediction. Our model is
designed with a weight-shared encoder and two task-oriented decoders and
optimized for the joint tasks. We introduce batch-Wasserstein (bW) loss and
construct a soft attention module to refine the distinctive visual region for
efficient saliency learning. For multitask optimization, it is always
challenging to obtain convergence of both tasks in the same epoch. We deal with
this problem by adopting `poly' loss weight and two phases of training. We
further propose a novel way to generate task-aware saliency map and scanpath of
the instruments on MICCAI robotic instrument segmentation dataset. Compared to
the state of the art segmentation and saliency models, our model outperforms
most of the evaluation metrics.
