  Humans quite frequently interact with conversational agents. The rapid
advancement in generative language modeling through neural networks has helped
advance the creation of intelligent conversational agents. Researchers
typically evaluate the output of their models through crowdsourced judgments,
but there are no established best practices for conducting such studies.
Moreover, it is unclear if cognitive biases in decision-making are affecting
crowdsourced workers' judgments when they undertake these tasks. To
investigate, we conducted a between-subjects study with 77 crowdsourced workers
to understand the role of cognitive biases, specifically anchoring bias, when
humans are asked to evaluate the output of conversational agents. Our results
provide insight into how best to evaluate conversational agents. We find
increased consistency in ratings across two experimental conditions may be a
result of anchoring bias. We also determine that external factors such as time
and prior experience in similar tasks have effects on inter-rater consistency.
