  In this work, we study two self-play training schemes, Chainer and Pool, and
show they lead to improved agent performance in Atari Pong compared to a
standard DQN agent -- trained against the built-in Atari opponent. To measure
agent performance, we define a robustness metric that captures how difficult it
is to learn a strategy that beats the agent's learned policy. Through playing
past versions of themselves, Chainer and Pool are able to target weaknesses in
their policies and improve their resistance to attack. Agents trained using
these methods score well on our robustness metric and can easily defeat the
standard DQN agent. We conclude by using linear probing to illuminate what
internal structures the different agents develop to play the game. We show that
training agents with Chainer or Pool leads to richer network activations with
greater predictive power to estimate critical game-state features compared to
the standard DQN agent.
