  This paper considers the challenges Large Language Models (LLMs) face when
reasoning over text that includes information involving uncertainty explicitly
quantified via probability values. This type of reasoning is relevant to a
variety of contexts ranging from everyday conversations to medical
decision-making. Despite improvements in the mathematical reasoning
capabilities of LLMs, they still exhibit significant difficulties when it comes
to probabilistic reasoning. To deal with this problem, we introduce the
Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically
designed to test the probabilistic reasoning capabilities of LLMs. We use BLInD
to find out the limitations of LLMs for tasks involving probabilistic
reasoning. In addition, we present several prompting strategies that map the
problem to different formal representations, including Python code,
probabilistic algorithms, and probabilistic logical programming. We conclude by
providing an evaluation of our methods on BLInD and an adaptation of a causal
reasoning question-answering dataset. Our empirical results highlight the
effectiveness of our proposed strategies for multiple LLMs.
