  Deep learning has emerged as a compelling solution to many NLP tasks with
remarkable performances. However, due to their opacity, such models are hard to
interpret and trust. Recent work on explaining deep models has introduced
approaches to provide insights toward the model's behaviour and predictions,
which are helpful for assessing the reliability of the model's predictions.
However, such methods do not improve the model's reliability. In this paper, we
aim to teach the model to make the right prediction for the right reason by
providing explanation training and ensuring the alignment of the model's
explanation with the ground truth explanation. Our experimental results on
multiple tasks and datasets demonstrate the effectiveness of the proposed
method, which produces more reliable predictions while delivering better
results compared to traditionally trained models.
