  Advances in deep learning systems have allowed large models to match or
surpass human accuracy on a number of skills such as image classification,
basic programming, and standardized test taking. As the performance of the most
capable models begin to saturate on tasks where humans already achieve high
accuracy, it becomes necessary to benchmark models on increasingly complex
abilities. One such task is forecasting the future outcome of events. In this
work we describe experiments using a novel dataset of real world events and
associated human predictions, an evaluation metric to measure forecasting
ability, and the accuracy of a number of different LLM based forecasting
designs on the provided dataset. Additionally, we analyze the performance of
the LLM forecasters against human predictions and find that models still
struggle to make accurate predictions about the future. Our follow-up
experiments indicate this is likely due to models' tendency to guess that most
events are unlikely to occur (which tends to be true for many prediction
datasets, but does not reflect actual forecasting abilities). We reflect on
next steps for developing a systematic and reliable approach to studying LLM
forecasting.
