  AI researchers employ not only the scientific method, but also methodology
from mathematics and engineering. However, the use of the scientific method -
specifically hypothesis testing - in AI is typically conducted in service of
engineering objectives. Growing interest in topics such as fairness and
algorithmic bias show that engineering-focused questions only comprise a subset
of the important questions about AI systems. This results in the AI Knowledge
Gap: the number of unique AI systems grows faster than the number of studies
that characterize these systems' behavior. To close this gap, we argue that the
study of AI could benefit from the greater inclusion of researchers who are
well positioned to formulate and test hypotheses about the behavior of AI
systems. We examine the barriers preventing social and behavioral scientists
from conducting such studies. Our diagnosis suggests that accelerating the
scientific study of AI systems requires new incentives for academia and
industry, mediated by new tools and institutions. To address these needs, we
propose a two-sided marketplace called TuringBox. On one side, AI contributors
upload existing and novel algorithms to be studied scientifically by others. On
the other side, AI examiners develop and post machine intelligence tasks
designed to evaluate and characterize algorithmic behavior. We discuss this
market's potential to democratize the scientific study of AI behavior, and thus
narrow the AI Knowledge Gap.
