  Question answering models can use rich knowledge sources -- up to one hundred
retrieved passages and parametric knowledge in the large-scale language model
(LM). Prior work assumes information in such knowledge sources is consistent
with each other, paying little attention to how models blend information stored
in their LM parameters with that from retrieved evidence documents. In this
paper, we simulate knowledge conflicts (i.e., where parametric knowledge
suggests one answer and different passages suggest different answers) and
examine model behaviors. We find retrieval performance heavily impacts which
sources models rely on, and current models mostly rely on non-parametric
knowledge in their best-performing settings. We discover a troubling trend that
contradictions among knowledge sources affect model confidence only marginally.
To address this issue, we present a new calibration study, where models are
discouraged from presenting any single answer when presented with multiple
conflicting answer candidates in retrieved evidences.
