  The advent of ChatGPT, a large language model-powered chatbot, has prompted
questions about its potential implications for traditional search engines. In
this study, we investigate the differences in user behavior when employing
search engines and chatbot tools for information-seeking tasks. We carry out a
randomized online experiment, dividing participants into two groups: one using
a ChatGPT-like tool and the other using a Google Search-like tool. Our findings
reveal that the ChatGPT group consistently spends less time on all tasks, with
no significant difference in overall task performance between the groups.
Notably, ChatGPT levels user search performance across different education
levels and excels in answering straightforward questions and providing general
solutions but falls short in fact-checking tasks. Users perceive ChatGPT's
responses as having higher information quality compared to Google Search,
despite displaying a similar level of trust in both tools. Furthermore,
participants using ChatGPT report significantly better user experiences in
terms of usefulness, enjoyment, and satisfaction, while perceived ease of use
remains comparable between the two tools. However, ChatGPT may also lead to
overreliance and generate or replicate misinformation, yielding inconsistent
results. Our study offers valuable insights for search engine management and
highlights opportunities for integrating chatbot technologies into search
engine designs.
