  Although chain-of-thought (CoT) prompting combined with language models has
achieved encouraging results on complex reasoning tasks, the naive greedy
decoding used in CoT prompting usually causes the repetitiveness and local
optimality. To address this shortcoming, ensemble-optimization tries to obtain
multiple reasoning paths to get the final answer assembly. However, current
ensemble-optimization methods either simply employ rule-based post-processing
such as \textit{self-consistency}, or train an additional model based on
several task-related human annotations to select the best one among multiple
reasoning paths, yet fail to generalize to realistic settings where the type of
input questions is unknown or the answer format of reasoning paths is unknown.
To avoid their limitations, we propose \textbf{Self-Agreement}, a generalizable
ensemble-optimization method applying in almost all scenarios where the type of
input questions and the answer format of reasoning paths may be known or
unknown. Self-agreement firstly samples from language model's decoder to
generate a \textit{diverse} set of reasoning paths, and subsequently prompts
the language model \textit{one more time} to determine the optimal answer by
selecting the most \textit{agreed} answer among the sampled reasoning paths.
Self-agreement simultaneously achieves remarkable performance on six public
reasoning benchmarks and superior generalization capabilities.
