  Chain-of-Thought (CoT) plays a crucial role in reasoning for math problem
solving. We conduct a comprehensive examination of methods for designing CoT,
comparing conventional natural language CoT with various program CoTs,
including the self-describing program, the comment-describing program, and the
non-describing program. Furthermore, we investigate the impact of programming
language on program CoTs, comparing Python and Wolfram Language. Through
extensive experiments on GSM8K, MATHQA, and SVAMP, we find that program CoTs
often have superior effectiveness in math problem solving. Notably, the best
performing combination with 30B parameters beats GPT-3.5-turbo by a significant
margin. The results show that self-describing program offers greater diversity
and thus can generally achieve higher performance. We also find that Python is
a better choice of language than Wolfram for program CoTs. The experimental
results provide a valuable guideline for future CoT designs that take into
account both programming language and coding style for further advancements.
Our datasets and code are publicly available.
