  Forecasting AI progress is essential to reducing uncertainty in order to
appropriately plan for research efforts on AI safety and AI governance. While
this is generally considered to be an important topic, little work has been
conducted on it and there is no published document that gives and objective
overview of the field. Moreover, the field is very diverse and there is no
published consensus regarding its direction. This paper describes the
development of a research agenda for forecasting AI progress which utilized the
Delphi technique to elicit and aggregate experts' opinions on what questions
and methods to prioritize. The results of the Delphi are presented; the
remainder of the paper follow the structure of these results, briefly reviewing
relevant literature and suggesting future work for each topic. Experts
indicated that a wide variety of methods should be considered for forecasting
AI progress. Moreover, experts identified salient questions that were both
general and completely unique to the problem of forecasting AI progress. Some
of the highest priority topics include the validation of (partially unresolved)
forecasts, how to make forecasting action-guiding and the quality of different
performance metrics. While statistical methods seem more promising, there is
also recognition that supplementing judgmental techniques can be quite
beneficial.
