  Subtle patterns in users' think-aloud (TA) verbalizations (i.e., utterances)
are shown to be telltale signs of user experience (UX) problems and used to
build artificial intelligence (AI) models or AI-assisted tools to help UX
evaluators identify UX problems automatically or semi-automatically. Despite
the potential of such verbalization patterns, they were uncovered with native
English speakers. As most people who speak English are non-native speakers, it
is important to investigate whether similar patterns exist in non-native
English speakers' TA verbalizations. As a first step to answer this question,
we conducted think-aloud usability testing with Chinese non-native English
speakers and native English speakers using three common TA protocols. We
compared their verbalizations and UX problems that they encountered to
understand the effects of language and TA protocols. Our findings show that
both language groups had similar amounts and proportions of verbalization
categories, encountered similar problems, and had similar verbalization
patterns that indicate UX problems. Furthermore, TA protocols did not
significantly affect the correlations between verbalizations and problems.
Based on the findings, we present three design implications for UX
practitioners and the design of AI-assisted analysis tools.
