  While the uptake of data-driven approaches for materials science and
chemistry is at an exciting, early stage, to realise the true potential of
machine learning models for successful scientific discovery, they must have
qualities beyond purely predictive power. The predictions and inner workings of
models should provide a certain degree of explainability by human experts,
permitting the identification of potential model issues or limitations,
building trust on model predictions and unveiling unexpected correlations that
may lead to scientific insights. In this work, we summarize applications of
interpretability and explainability techniques for materials science and
chemistry and discuss how these techniques can improve the outcome of
scientific studies. We discuss various challenges for interpretable machine
learning in materials science and, more broadly, in scientific settings. In
particular, we emphasize the risks of inferring causation or reaching
generalization by purely interpreting machine learning models and the need of
uncertainty estimates for model explanations. Finally, we showcase a number of
exciting developments in other fields that could benefit interpretability in
material science and chemistry problems.
