  In this research study, we empirically investigate the effect of sampling
temperature on the performance of Large Language Models (LLMs) on various
problem-solving tasks. We created a multiple-choice question-and-answer (MCQA)
exam by randomly sampling problems from standard LLM benchmarks. Then, we used
nine popular LLMs with five prompt-engineering techniques to solve the MCQA
problems while increasing the sampling temperature from 0.0 to 1.6. Despite
anecdotal reports to the contrary, our empirical results indicate that changes
in temperature from 0.0 to 1.0 do not have a statistically significant impact
on LLM performance for problem-solving tasks. In addition, these results appear
to generalize across LLMs, prompt-engineering techniques, and problem domains.
All code, data, and supplemental materials are available on GitHub at:
https://github.com/matthewrenze/jhu-llm-temperature
