  Deep reinforcement learning has learned to play many games well, but failed
on others. To better characterize the modes and reasons of failure of deep
reinforcement learners, we test the widely used Asynchronous Actor-Critic (A2C)
algorithm on four deceptive games, which are specially designed to provide
challenges to game-playing agents. These games are implemented in the General
Video Game AI framework, which allows us to compare the behavior of
reinforcement learning-based agents with planning agents based on tree search.
We find that several of these games reliably deceive deep reinforcement
learners, and that the resulting behavior highlights the shortcomings of the
learning algorithm. The particular ways in which agents fail differ from how
planning-based agents fail, further illuminating the character of these
algorithms. We propose an initial typology of deceptions which could help us
better understand pitfalls and failure modes of (deep) reinforcement learning.
