  Self-trained autonomous agents developed using machine learning are showing
great promise in a variety of control settings, perhaps most remarkably in
applications involving autonomous vehicles. The main challenge associated with
self-learned agents in the form of deep neural networks, is their black-box
nature: it is impossible for humans to interpret deep neural networks.
Therefore, humans cannot directly interpret the actions of deep neural network
based agents, or foresee their robustness in different scenarios. In this work,
we demonstrate a method for probing which concepts self-learning agents
internalise in the course of their training. For demonstration, we use a chess
playing agent in a fast and light environment developed specifically to be
suitable for research groups without access to enormous computational resources
or machine learning models.
