  Machine learning models have had discernible achievements in a myriad of
applications. However, most of these models are black-boxes, and it is obscure
how the decisions are made by them. This makes the models unreliable and
untrustworthy. To provide insights into the decision making processes of these
models, a variety of traditional interpretable models have been proposed.
Moreover, to generate more human-friendly explanations, recent work on
interpretability tries to answer questions related to causality such as "Why
does this model makes such decisions?" or "Was it a specific feature that
caused the decision made by the model?". In this work, models that aim to
answer causal questions are referred to as causal interpretable models. The
existing surveys have covered concepts and methodologies of traditional
interpretability. In this work, we present a comprehensive survey on causal
interpretable models from the aspects of the problems and methods. In addition,
this survey provides in-depth insights into the existing evaluation metrics for
measuring interpretability, which can help practitioners understand for what
scenarios each evaluation metric is suitable.
