  Machine learned potentials are becoming a popular tool to define an effective
energy model for complex systems, either incorporating electronic structure
effects at the atomistic resolution, or effectively renormalizing part of the
atomistic degrees of freedom at a coarse-grained resolution. One of the main
criticisms to machine learned potentials is that the energy inferred by the
network is not as interpretable as in more traditional approaches where a
simpler functional form is used. Here we address this problem by extending
tools recently proposed in the nascent field of Explainable Artificial
Intelligence (XAI) to coarse-grained potentials based on graph neural networks
(GNN). We demonstrate the approach on three different coarse-grained systems
including two fluids (methane and water) and the protein NTL9. On these
examples, we show that the neural network potentials can be in practice
decomposed in relevance contributions to different orders, that can be directly
interpreted and provide physical insights on the systems of interest.
