  To foster usefulness and accountability of machine learning (ML), it is
essential to explain a model's decisions in addition to evaluating its
performance. Accordingly, the field of explainable artificial intelligence
(XAI) has resurfaced as a topic of active research, offering approaches to
address the "how" and "why" of automated decision-making. Within this domain,
counterfactual explanations (CFEs) have gained considerable traction as a
psychologically grounded approach to generate post-hoc explanations. To do so,
CFEs highlight what changes to a model's input would have changed its
prediction in a particular way. However, despite the introduction of numerous
CFE approaches, their usability has yet to be thoroughly validated at the human
level. Thus, to advance the field of XAI, we introduce the Alien Zoo, an
engaging, web-based and game-inspired experimental framework. The Alien Zoo
provides the means to evaluate usability of CFEs for gaining new knowledge from
an automated system, targeting novice users in a domain-general context. As a
proof of concept, we demonstrate the practical efficacy and feasibility of this
approach in a user study. Our results suggest that users benefit from receiving
CFEs compared to no explanation, both in terms of objective performance in the
proposed iterative learning task, and subjective usability. With this work, we
aim to equip research groups and practitioners with the means to easily run
controlled and well-powered user studies to complement their otherwise often
more technology-oriented work. Thus, in the interest of reproducible research,
we provide the entire code, together with the underlying models and user data.
