  How can intelligent machines assess their competencies in completing tasks?
This question has come into focus for autonomous systems that algorithmically
reason and make decisions under uncertainty. It is argued here that machine
self-confidence - a form of meta-reasoning based on self-assessments of an
agent's knowledge about the state of the world and itself, as well as its
ability to reason about and execute tasks - leads to many eminently computable
and useful competency indicators for such agents. This paper presents a
culmination of work on this concept in the form of a computational framework
called Factorized Machine Self-confidence (FaMSeC), which provides a holistic
engineering-focused description of factors driving an algorithmic
decision-making process, including: outcome assessment, solver quality, model
quality, alignment quality, and past experience. In FaMSeC, self confidence
indicators are derived from hierarchical `problem-solving statistics' embedded
within broad classes of probabilistic decision-making algorithms such as Markov
decision processes. The problem-solving statistics are obtained by evaluating
and grading probabilistic exceedance margins with respect to given competency
standards, which are specified for each of the various decision-making
competency factors by the informee (e.g. a non-expert user or an expert system
designer). This approach allows `algorithmic goodness of fit' evaluations to be
easily incorporated into the design of many kinds of autonomous agents in the
form of human-interpretable competency self-assessment reports. Detailed
descriptions and application examples for a Markov decision process agent show
how two of the FaMSeC factors (outcome assessment and solver quality) can be
computed and reported for a range of possible tasking contexts through novel
use of meta-utility functions, behavior simulations, and surrogate prediction
models.
