  Artificial Intelligence (AI) is about making computers that do the sorts of
things that minds can do, and as we progress towards this goal, we tend to
increasingly delegate human tasks to machines. However, AI systems usually do
these tasks with an unusual imbalance of insight and understanding: new, deeper
insights are present, yet many important qualities that a human mind would have
previously brought to the activity are utterly absent. Therefore, it is crucial
to ask which features of minds have we replicated, which are missing, and if
that matters. One core feature that humans bring to tasks, when dealing with
the ambiguity, emergent knowledge, and social context presented by the world,
is reflection. Yet this capability is utterly missing from current mainstream
AI. In this paper we ask what reflective AI might look like. Then, drawing on
notions of reflection in complex systems, cognitive science, and agents, we
sketch an architecture for reflective AI agents, and highlight ways forward.
