  The surprisingly likely criterion in the seminal work of Prelec (the Bayesian
Truth Serum) guarantees truthfulness in a game-theoretic multi-agent setting,
by rewarding rational agents to maximise the expected information gain with
their answers w.r.t. their probabilistic beliefs. We investigate the relevance
of a similar criterion for responses of LLMs. We hypothesize that if the
surprisingly likely criterion works in LLMs, under certain conditions, the
responses that maximize the reward under this criterion should be more accurate
than the responses that only maximize the posterior probability. Using
benchmarks including the TruthfulQA benchmark and using openly available LLMs:
GPT-2 and LLaMA-2, we show that the method indeed improves the accuracy
significantly (for example, upto 24 percentage points aggregate improvement on
TruthfulQA and upto 70 percentage points improvement on individual categories
of questions).
