  We solve university level probability and statistics questions by program
synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on
code. We transform course problems from MIT's 18.05 Introduction to Probability
and Statistics and Harvard's STAT110 Probability into programming tasks. We
then execute the generated code to get a solution. Since these course questions
are grounded in probability, we often aim to have Codex generate probabilistic
programs that simulate a large number of probabilistic dependencies to compute
its solution. Our approach requires prompt engineering to transform the
question from its original form to an explicit, tractable form that results in
a correct program and solution. To estimate the amount of work needed to
translate an original question into its tractable form, we measure the
similarity between original and transformed questions. Our work is the first to
introduce a new dataset of university-level probability and statistics problems
and solve these problems in a scalable fashion using the program synthesis
capabilities of large language models.
