  Artificial intelligence (AI) has revolutionized decision-making processes and
systems throughout society and, in particular, has emerged as a significant
technology in high-impact scenarios of national interest. Yet, despite AI's
impressive predictive capabilities in controlled settings, it still suffers
from a range of practical setbacks preventing its widespread use in various
critical scenarios. In particular, it is generally unclear if a given AI
system's predictions can be trusted by decision-makers in downstream
applications. To address the need for more transparent, robust, and trustworthy
AI systems, a suite of tools has been developed to quantify the uncertainty of
AI predictions and, more generally, enable AI to "self-assess" the reliability
of its predictions. In this manuscript, we categorize methods for AI
self-assessment along several key dimensions and provide guidelines for
selecting and designing the appropriate method for a practitioner's needs. In
particular, we focus on uncertainty estimation techniques that consider the
impact of self-assessment on the choices made by downstream decision-makers and
on the resulting costs and benefits of decision outcomes. To demonstrate the
utility of our methodology for self-assessment design, we illustrate its use
for two realistic national-interest scenarios. This manuscript is a practical
guide for machine learning engineers and AI system users to select the ideal
self-assessment techniques for each problem.
