  Machine Learning models become increasingly proficient in complex tasks.
However, even for experts in the field, it can be difficult to understand what
the model learned. This hampers trust and acceptance, and it obstructs the
possibility to correct the model. There is therefore a need for transparency of
machine learning models. The development of transparent classification models
has received much attention, but there are few developments for achieving
transparent Reinforcement Learning (RL) models. In this study we propose a
method that enables a RL agent to explain its behavior in terms of the expected
consequences of state transitions and outcomes. First, we define a translation
of states and actions to a description that is easier to understand for human
users. Second, we developed a procedure that enables the agent to obtain the
consequences of a single action, as well as its entire policy. The method
calculates contrasts between the consequences of a policy derived from a user
query, and of the learned policy of the agent. Third, a format for generating
explanations was constructed. A pilot survey study was conducted to explore
preferences of users for different explanation properties. Results indicate
that human users tend to favor explanations about policy rather than about
single actions.
