  We reframe the analysis of progress in AI by incorporating into an overall
framework both the task performance of a system, and the time and resource
costs incurred in the development and deployment of the system. These costs
include: data, expert knowledge, human oversight, software resources, computing
cycles, hardware and network facilities, and (what kind of) time. These costs
are distributed over the life cycle of the system, and may place differing
demands on different developers and users. The multidimensional performance and
cost space we present can be collapsed to a single utility metric that measures
the value of the system for different stakeholders. Even without a single
utility function, AI advances can be generically assessed by whether they
expand the Pareto surface. We label these types of costs as neglected
dimensions of AI progress, and explore them using four case studies: Alpha*
(Go, Chess, and other board games), ALE (Atari games), ImageNet (Image
classification) and Virtual Personal Assistants (Siri, Alexa, Cortana, and
Google Assistant). This broader model of progress in AI will lead to novel ways
of estimating the potential societal use and impact of an AI system, and the
establishment of milestones for future progress.
