  The ambiguity of the decision-making process has been pointed out as the main
obstacle to applying the deep learning-based method in a practical way in spite
of its outstanding performance. Interpretability could guarantee the confidence
of deep learning system, therefore it is particularly important in the medical
field. In this study, a novel deep network is proposed to explain the
diagnostic decision with visual pointing map and diagnostic sentence justifying
result simultaneously. For the purpose of increasing the accuracy of sentence
generation, a visual word constraint model is devised in training justification
generator. To verify the proposed method, comparative experiments were
conducted on the problem of the diagnosis of breast masses. Experimental
results demonstrated that the proposed deep network could explain diagnosis
more accurately with various textual justifications.
