  ChatGPT is a popular information system (over 1 billion visits in August
2023) that can generate natural language responses to user queries. It is
important to study the quality and equity of its responses on health-related
topics, such as vaccination, as they may influence public health
decision-making. We use the Vaccine Hesitancy Scale (VHS) proposed by Shapiro
et al.1 to measure the hesitancy of ChatGPT responses in English, Spanish, and
French. We find that: (a) ChatGPT responses indicate less hesitancy than those
reported for human respondents in past literature; (b) ChatGPT responses vary
significantly across languages, with English responses being the most hesitant
on average and Spanish being the least; (c) ChatGPT responses are largely
consistent across different model parameters but show some variations across
the scale factors (vaccine competency, risk). Results have implications for
researchers interested in evaluating and improving the quality and equity of
health-related web information.
