  Despite the remarkable performance, Deep Neural Networks (DNNs) behave as
black-boxes hindering user trust in Artificial Intelligence (AI) systems.
Research on opening black-box DNN can be broadly categorized into post-hoc
methods and inherently interpretable DNNs. While many surveys have been
conducted on post-hoc interpretation methods, little effort is devoted to
inherently interpretable DNNs. This paper provides a review of existing methods
to develop DNNs with intrinsic interpretability, with a focus on Convolutional
Neural Networks (CNNs). The aim is to understand the current progress towards
fully interpretable DNNs that can cater to different interpretation
requirements. Finally, we identify gaps in current work and suggest potential
research directions.
