  With the long term accumulation of high quality educational data, artificial
intelligence has shown excellent performance in knowledge tracing. However, due
to the lack of interpretability and transparency of some algorithms, this
approach will result in reduced stakeholder trust and a decreased acceptance of
intelligent decisions. Therefore, algorithms need to achieve high accuracy, and
users need to understand the internal operating mechanism and provide reliable
explanations for decisions. This paper thoroughly analyzes the interpretability
of KT algorithms. First, the concepts and common methods of explainable
artificial intelligence and knowledge tracing are introduced. Next, explainable
knowledge tracing models are classified into two categories: transparent models
and black box models. Then, the interpretable methods used are reviewed from
three stages: ante hoc interpretable methods, post hoc interpretable methods,
and other dimensions. It is worth noting that current evaluation methods for
explainable knowledge tracing are lacking. Hence, contrast and deletion
experiments are conducted to explain the prediction results of the deep
knowledge tracing model on the ASSISTment2009 by using three XAI methods.
Moreover, this paper offers some insights into evaluation methods from the
perspective of educational stakeholders. This paper provides a detailed and
comprehensive review of the research on explainable knowledge tracing, aiming
to offer some basis and inspiration for researchers interested in the
interpretability of knowledge tracing.
