  Artificial Intelligence (AI), with ChatGPT as a prominent example, has
recently taken center stage in various domains including higher education,
particularly in Computer Science and Engineering (CSE). The AI revolution
brings both convenience and controversy, offering substantial benefits while
lacking formal guidance on their application. The primary objective of this
work is to comprehensively analyze the pedagogical potential of ChatGPT in CSE
education, understanding its strengths and limitations from the perspectives of
educators and learners. We employ a systematic approach, creating a diverse
range of educational practice problems within CSE field, focusing on various
subjects such as data science, programming, AI, machine learning, networks, and
more. According to our examinations, certain question types, like conceptual
knowledge queries, typically do not pose significant challenges to ChatGPT, and
thus, are excluded from our analysis. Alternatively, we focus our efforts on
developing more in-depth and personalized questions and project-based tasks.
These questions are presented to ChatGPT, followed by interactions to assess
its effectiveness in delivering complete and meaningful responses. To this end,
we propose a comprehensive five-factor reliability analysis framework to
evaluate the responses. This assessment aims to identify when ChatGPT excels
and when it faces challenges. Our study concludes with a correlation analysis,
delving into the relationships among subjects, task types, and limiting
factors. This analysis offers valuable insights to enhance ChatGPT's utility in
CSE education, providing guidance to educators and students regarding its
reliability and efficacy.
