  Machine learning (ML) is a promising tool for the detection of phases of
matter. However, ML models are also known for their black-box construction,
which hinders understanding of what they learn from the data and makes their
application to novel data risky. Moreover, the central challenge of ML is to
ensure its good generalization abilities, i.e., good performance on data
outside the training set. Here, we show how the informed use of an
interpretability method called class activation mapping (CAM), and the analysis
of the latent representation of the data with the principal component analysis
(PCA) can increase trust in predictions of a neural network (NN) trained to
classify quantum phases. In particular, we show that we can ensure better
out-of-distribution generalization in the complex classification problem by
choosing such an NN that, in the simplified version of the problem, learns a
known characteristic of the phase. We show this on an example of the
topological Su-Schrieffer-Heeger (SSH) model with and without disorder, which
turned out to be surprisingly challenging for NNs trained in a supervised way.
This work is an example of how the systematic use of interpretability methods
can improve the performance of NNs in scientific problems.
