  Recent applications of autonomous agents and robots, such as self-driving
cars, scenario-based trainers, exploration robots, and service robots have
brought attention to crucial trust-related challenges associated with the
current generation of artificial intelligence (AI) systems. AI systems based on
the connectionist deep learning neural network approach lack capabilities of
explaining their decisions and actions to others, despite their great
successes. Without symbolic interpretation capabilities, they are black boxes,
which renders their decisions or actions opaque, making it difficult to trust
them in safety-critical applications. The recent stance on the explainability
of AI systems has witnessed several approaches on eXplainable Artificial
Intelligence (XAI); however, most of the studies have focused on data-driven
XAI systems applied in computational sciences. Studies addressing the
increasingly pervasive goal-driven agents and robots are still missing. This
paper reviews approaches on explainable goal-driven intelligent agents and
robots, focusing on techniques for explaining and communicating agents
perceptual functions (example, senses, and vision) and cognitive reasoning
(example, beliefs, desires, intention, plans, and goals) with humans in the
loop. The review highlights key strategies that emphasize transparency,
understandability, and continual learning for explainability. Finally, the
paper presents requirements for explainability and suggests a roadmap for the
possible realization of effective goal-driven explainable agents and robots.
