ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions

Abstract

This research paper aims to analyze the strengths and weaknesses associated with the utilization of ChatGPT as an educational tool in the context of undergraduate computer science education. ChatGPT's usage in tasks such as solving assignments and exams has the potential to undermine students' learning outcomes and compromise academic integrity. This study adopts a quantitative approach to demonstrate the notable unreliability of ChatGPT in providing accurate answers to a wide range of questions within the field of undergraduate computer science. While the majority of existing research has concentrated on assessing the performance of Large Language Models in handling programming assignments, our study adopts a more comprehensive approach. Specifically, we evaluate various types of questions such as true/false, multi-choice, multi-select, short answer, long answer, design-based, and coding-related questions. Our evaluation highlights the potential consequences of students excessively relying on ChatGPT for the completion of assignments and exams, including self-sabotage. We conclude with a discussion on how can students and instructors constructively use ChatGPT and related tools to enhance the quality of instruction and the overall student experience.