  Large language models primarily rely on inductive reasoning for decision
making. This results in unreliable decisions when applied to real-world tasks
that often present incomplete contexts and conditions. Thus, accurate
probability estimation and appropriate interpretations are required to enhance
decision-making reliability. In this paper, we propose a Bayesian inference
framework called BIRD for large language models. BIRD provides controllable and
interpretable probability estimation for model decisions, based on abductive
factors, LLM entailment, as well as learnable deductive Bayesian modeling.
Experiments show that BIRD produces probability estimations that align with
human judgments over 65% of the time using open-sourced Llama models,
outperforming the state-of-the-art GPT-4 by 35%. We also show that BIRD can be
directly used for trustworthy decision making on many real-world applications.
