  Motivated by the interpretability question in ML models as a crucial element
for the successful deployment of AI systems, this paper focuses on rule
extraction as a means for neural networks interpretability. Through a
systematic literature review, different approaches for extracting rules from
feedforward neural networks, an important block in deep learning models, are
identified and explored. The findings reveal a range of methods developed for
over two decades, mostly suitable for shallow neural networks, with recent
developments to meet deep learning models' challenges. Rules offer a
transparent and intuitive means of explaining neural networks, making this
study a comprehensive introduction for researchers interested in the field.
While the study specifically addresses feedforward networks with supervised
learning and crisp rules, future work can extend to other network types,
machine learning methods, and fuzzy rule extraction.
