  How does the brain control attention? The Attention Schema Theory suggests
that the brain explicitly models its state of attention, termed an attention
schema, for its control. However, it remains unclear under which circumstances
an attention schema is computationally useful, and whether it can emerge in a
learning system without hard-wiring. To address these questions, we trained a
reinforcement learning agent with attention to track and catch a ball in a
noisy environment. Crucially, the agent had additional resources that it could
freely use. We asked under which conditions these additional resources develop
an attention schema to track attention. We found that the more uncertain the
agent was about the location of its attentional window, the more it benefited
from these additional resources, which developed an attention schema. Together,
these results indicate that an attention schema emerges in simple learning
systems where attention is important and difficult to track.
