  A diverse array of reasoning strategies has been proposed to elicit the
capabilities of large language models. However, in this paper, we point out
that traditional evaluations which focus solely on performance metrics miss a
key factor: the increased effectiveness due to additional compute. By
overlooking this aspect, a skewed view of strategy efficiency is often
presented. This paper introduces a framework that incorporates the compute
budget into the evaluation, providing a more informative comparison that takes
into account both performance metrics and computational cost. In this
budget-aware perspective, we find that complex reasoning strategies often don't
surpass simpler baselines purely due to algorithmic ingenuity, but rather due
to the larger computational resources allocated. When we provide a simple
baseline like chain-of-thought self-consistency with comparable compute
resources, it frequently outperforms reasoning strategies proposed in the
literature. In this scale-aware perspective, we find that unlike
self-consistency, certain strategies such as multi-agent debate or Reflexion
can become worse if more compute budget is utilized.
