  Solar flares are extremely energetic phenomena in our Solar System. Their
impulsive, often drastic radiative increases, in particular at short
wavelengths, bring immediate impacts that motivate solar physics and space
weather research to understand solar flares to the point of being able to
forecast them. As data and algorithms improve dramatically, questions must be
asked concerning how well the forecasting performs; crucially, we must ask how
to rigorously measure performance in order to critically gauge any
improvements. Building upon earlier-developed methodology (Barnes et al, 2016,
Paper I), international representatives of regional warning centers and
research facilities assembled in 2017 at the Institute for Space-Earth
Environmental Research, Nagoya University, Japan to - for the first time -
directly compare the performance of operational solar flare forecasting
methods. Multiple quantitative evaluation metrics are employed, with focus and
discussion on evaluation methodologies given the restrictions of operational
forecasting. Numerous methods performed consistently above the "no skill"
level, although which method scored top marks is decisively a function of flare
event definition and the metric used; there was no single winner. Following in
this paper series we ask why the performances differ by examining
implementation details (Leka et al. 2019, Paper III), and then we present a
novel analysis method to evaluate temporal patterns of forecasting errors in
(Park et al. 2019, Paper IV). With these works, this team presents a
well-defined and robust methodology for evaluating solar flare forecasting
methods in both research and operational frameworks, and today's performance
benchmarks against which improvements and new methods may be compared.
