  The increased deployment of LMs for real-world tasks involving knowledge and
facts makes it important to understand model epistemology: what LMs think they
know, and how their attitudes toward that knowledge are affected by language
use in their inputs. Here, we study an aspect of model epistemology: how
epistemic markers of certainty, uncertainty, or evidentiality like "I'm sure
it's", "I think it's", or "Wikipedia says it's" affect models, and whether they
contribute to model failures. We develop a typology of epistemic markers and
inject 50 markers into prompts for question answering. We find that LMs are
highly sensitive to epistemic markers in prompts, with accuracies varying more
than 80%. Surprisingly, we find that expressions of high certainty result in a
7% decrease in accuracy as compared to low certainty expressions; similarly,
factive verbs hurt performance, while evidentials benefit performance. Our
analysis of a popular pretraining dataset shows that these markers of
uncertainty are associated with answers on question-answering websites, while
markers of certainty are associated with questions. These associations may
suggest that the behavior of LMs is based on mimicking observed language use,
rather than truly reflecting epistemic uncertainty.
