  Indecipherable black boxes are common in machine learning (ML), but
applications increasingly require explainable artificial intelligence (XAI).
The core of XAI is to establish transparent and interpretable data-driven
algorithms. This work provides concrete tools for XAI in situations where prior
knowledge must be encoded and untrustworthy inferences flagged. We use the
"learn to optimize" (L2O) methodology wherein each inference solves a
data-driven optimization problem. Our L2O models are straightforward to
implement, directly encode prior knowledge, and yield theoretical guarantees
(e.g. satisfaction of constraints). We also propose use of interpretable
certificates to verify whether model inferences are trustworthy. Numerical
examples are provided in the applications of dictionary-based signal recovery,
CT imaging, and arbitrage trading of cryptoassets. Code and additional
documentation can be found at https://xai-l2o.research.typal.academy.
