  Large language models (LLMs) such as GPT-4 have exhibited remarkable
performance in a variety of tasks, but this strong performance often comes with
the high expense of using paid API services. In this paper, we are motivated to
study building an LLM cascade to save the cost of using LLMs, particularly for
performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline
follows the intuition that simpler questions can be addressed by a weaker but
more affordable LLM, whereas only the challenging questions necessitate the
stronger and more expensive LLM. To realize this decision-making, we consider
the "answer consistency" of the weaker LLM as a signal of the question
difficulty and propose several methods for the answer sampling and consistency
checking, including one leveraging a mixture of two thought representations
(i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six
reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and
stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can
achieve performance comparable to using solely the stronger LLM but require
only 40% of its cost.
