  This study explores the behavior of machine learning-based flare forecasting
models deployed in a simulated operational environment. Using Georgia State
University's Space Weather Analytics for Solar Flares benchmark dataset (Angryk
et al. 2020a,b), we examine the impacts of training methodology and the solar
cycle on decision tree, support vector machine, and multilayer perceptron
performance. We implement our classifiers using three temporal training
windows: stationary, rolling, and expanding. The stationary window trains
models using a single set of data available before the first forecasting
instance, which remains constant throughout the solar cycle. The rolling window
trains models using data from a constant time interval before the forecasting
instance, which moves with the solar cycle. Finally, the expanding window
trains models using all available data before the forecasting instance. For
each window, a number of input features (1, 5, 10, 25, 50, 120) and temporal
sizes (5, 8, 11, 14, 17, 20 months) were tested. To our surprise, we found that
for a 20-month window, skill scores were comparable regardless of the window
type, feature count, and classifier selected. Furthermore, reducing the size of
this window only marginally decreased stationary and rolling window
performance. This implies that, given enough data, a stationary window can be
chosen over other window types, eliminating the need for model retraining.
Lastly, a moderately strong positive correlation was found to exist between a
model's false positive rate and the solar X-ray background flux. This suggests
that the solar cycle phase has a considerable influence on forecasting.
