  Q&A platforms have been crucial for the online help-seeking behavior of
programmers. However, the recent popularity of ChatGPT is altering this trend.
Despite this popularity, no comprehensive study has been conducted to evaluate
the characteristics of ChatGPT's answers to programming questions. To bridge
the gap, we conducted the first in-depth analysis of ChatGPT answers to 517
programming questions on Stack Overflow and examined the correctness,
consistency, comprehensiveness, and conciseness of ChatGPT answers.
Furthermore, we conducted a large-scale linguistic analysis, as well as a user
study, to understand the characteristics of ChatGPT answers from linguistic and
human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect
information and 77% are verbose. Nonetheless, our user study participants still
preferred ChatGPT answers 35% of the time due to their comprehensiveness and
well-articulated language style. However, they also overlooked the
misinformation in the ChatGPT answers 39% of the time. This implies the need to
counter misinformation in ChatGPT answers to programming questions and raise
awareness of the risks associated with seemingly correct answers.
