  Although pretrained language models (PTLMs) contain significant amounts of
world knowledge, they can still produce inconsistent answers to questions when
probed, even after specialized training. As a result, it can be hard to
identify what the model actually "believes" about the world, making it
susceptible to inconsistent behavior and simple errors. Our goal is to reduce
these problems. Our approach is to embed a PTLM in a broader system that also
includes an evolving, symbolic memory of beliefs -- a BeliefBank -- that
records but then may modify the raw PTLM answers. We describe two mechanisms to
improve belief consistency in the overall system. First, a reasoning component
-- a weighted MaxSAT solver -- revises beliefs that significantly clash with
others. Second, a feedback component issues future queries to the PTLM using
known beliefs as context. We show that, in a controlled experimental setting,
these two mechanisms result in more consistent beliefs in the overall system,
improving both the accuracy and consistency of its answers over time. This is
significant as it is a first step towards PTLM-based architectures with a
systematic notion of belief, enabling them to construct a more coherent picture
of the world, and improve over time without model retraining.
