  In recent years there has been increased use of machine learning (ML)
techniques within mathematics, including symbolic computation where it may be
applied safely to optimise or select algorithms. This paper explores whether
using explainable AI (XAI) techniques on such ML models can offer new insight
for symbolic computation, inspiring new implementations within computer algebra
systems that do not directly call upon AI tools. We present a case study on the
use of ML to select the variable ordering for cylindrical algebraic
decomposition. It has already been demonstrated that ML can make the choice
well, but here we show how the SHAP tool for explainability can be used to
inform new heuristics of a size and complexity similar to those human-designed
heuristics currently commonly used in symbolic computation.
