  The surge in black-box AI models has prompted the need to explain the
internal mechanism and justify their reliability, especially in high-stakes
applications, such as healthcare and autonomous driving. Due to the lack of a
rigorous definition of explainable AI (XAI), a plethora of research related to
explainability, interpretability, and transparency has been developed to
explain and analyze the model from various perspectives. Consequently, with an
exhaustive list of papers, it becomes challenging to have a comprehensive
overview of XAI research from all aspects. Considering the popularity of neural
networks in AI research, we narrow our focus to a specific area of XAI
research: gradient based explanations, which can be directly adopted for neural
network models. In this review, we systematically explore gradient based
explanation methods to date and introduce a novel taxonomy to categorize them
into four distinct classes. Then, we present the essence of technique details
in chronological order and underscore the evolution of algorithms. Next, we
introduce both human and quantitative evaluations to measure algorithm
performance. More importantly, we demonstrate the general challenges in XAI and
specific challenges in gradient based explanations. We hope that this survey
can help researchers understand state-of-the-art progress and their
corresponding disadvantages, which could spark their interest in addressing
these issues in future work.
