  Despite their impact on the society, deep neural networks are often regarded
as black-box models due to their intricate structures and the absence of
explanations for their decisions. This opacity poses a significant challenge to
AI systems wider adoption and trustworthiness. This thesis addresses this issue
by contributing to the field of eXplainable AI, focusing on enhancing the
interpretability of deep neural networks. The core contributions lie in
introducing novel techniques aimed at making these networks more interpretable
by leveraging an analysis of their inner workings. Specifically, the
contributions are threefold. Firstly, the thesis introduces designs for
self-explanatory deep neural networks, such as the integration of external
memory for interpretability purposes and the usage of prototype and
constraint-based layers across several domains. Secondly, this research delves
into novel investigations on neurons within trained deep neural networks,
shedding light on overlooked phenomena related to their activation values.
Lastly, the thesis conducts an analysis of the application of explanatory
techniques in the field of visual analytics, exploring the maturity of their
adoption and the potential of these systems to convey explanations to users
effectively.
