  Deep neural networks (DNNs) can be useful within the marine robotics field,
but their utility value is restricted by their black-box nature. Explainable
artificial intelligence methods attempt to understand how such black-boxes make
their decisions. In this work, linear model trees (LMTs) are used to
approximate the DNN controlling an autonomous surface vessel (ASV) in a
simulated environment and then run in parallel with the DNN to give
explanations in the form of feature attributions in real-time. How well a model
can be understood depends not only on the explanation itself, but also on how
well it is presented and adapted to the receiver of said explanation. Different
end-users may need both different types of explanations, as well as different
representations of these. The main contributions of this work are (1)
significantly improving both the accuracy and the build time of a greedy
approach for building LMTs by introducing ordering of features in the splitting
of the tree, (2) giving an overview of the characteristics of the
seafarer/operator and the developer as two different end-users of the agent and
receiver of the explanations, and (3) suggesting a visualization of the docking
agent, the environment, and the feature attributions given by the LMT for when
the developer is the end-user of the system, and another visualization for when
the seafarer or operator is the end-user, based on their different
characteristics.
