  ChatGPT is one of the most popular language models which achieve amazing
performance on various natural language tasks. Consequently, there is also an
urgent need to detect the texts generated ChatGPT from human written. One of
the extensively studied methods trains classification models to distinguish
both. However, existing studies also demonstrate that the trained models may
suffer from distribution shifts (during test), i.e., they are ineffective to
predict the generated texts from unseen language tasks or topics. In this work,
we aim to have a comprehensive investigation on these methods' generalization
behaviors under distribution shift caused by a wide range of factors, including
prompts, text lengths, topics, and language tasks. To achieve this goal, we
first collect a new dataset with human and ChatGPT texts, and then we conduct
extensive studies on the collected dataset. Our studies unveil insightful
findings which provide guidance for developing future methodologies or data
collection strategies for ChatGPT detection.
