  Assessing and understanding intelligent agents is a difficult task for users
that lack an AI background. A relatively new area, called "Explainable AI," is
emerging to help address this problem, but little is known about how users
would forage through information an explanation system might offer. To inform
the development of Explainable AI systems, we conducted a formative study,
using the lens of Information Foraging Theory, into how experienced users
foraged in the domain of StarCraft to assess an agent. Our results showed that
participants faced difficult foraging problems. These foraging problems caused
participants to entirely miss events that were important to them, reluctantly
choose to ignore actions they did not want to ignore, and bear high cognitive,
navigation, and information costs to access the information they needed.
