  Recently, ChatGPT has attracted great attention, as it can generate fluent
and high-quality responses to human inquiries. Several prior studies have shown
that ChatGPT attains remarkable generation ability compared with existing
models. However, the quantitative analysis of ChatGPT's understanding ability
has been given little attention. In this report, we explore the understanding
ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and
comparing it with 4 representative fine-tuned BERT-style models. We find that:
1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT
outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT
achieves comparable performance compared with BERT on sentiment analysis and
question-answering tasks. Additionally, by combining some advanced prompting
strategies, we show that the understanding ability of ChatGPT can be further
improved.
