Document Type

Working Paper

Publication Date

3-11-2023

Abstract

AI (Artificial Intelligence) Generative Models have attracted great attention in recent years. Generative models can be used to create new articles, visual arts, music composition, even computer programs from English specifications. Among all generative models, ChatGPT is becoming one of the most well-known since its public announcement in November 2022. GPT means {\it Generative Pre-trained Transformer}. ChatGPT is an online program that can interact with human users in text formats and is able to answer questions in many topics, including computer programming. Many computer programmers, including students and professionals, are considering the use of ChatGPT as an aid. The quality of ChatGPT's aid is therefore of interest. To shed some light on this quality, this article presents conversations between the authors and ChatGPT. These conversations are analyzed to understand ChatGPT's ability to answer questions related to computer programming. We consider questions that may appear in C programming at the introductory levels. These questions are classified into different categories: (1) facts that can be looked up from documentations, (2) extension and derivation of facts, (3) simple computer programs, (4) extension of simple computer programs, (5) debugging, and (6) integration of discrete mathematics and C programming. This study discovers that ChatGPT can provide correct answers in many cases, but also make mistakes in the others. For a sequence of related questions, ChatGPT may provide inconsistent answers and exhibits self contradiction. It is also possible that ChatGPT gives a program with {\it security vulnerability}. ChatGPT may give different answers when the same questions are asked again. This article reports ChatGPT's answers captured in February and March 2023.

Comments

Author Posting © The Author, 2023. This is a working paper, which can be accessed on Figshare: https://doi.org/10.6084/m9.figshare.22257274

Creative Commons License

Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.

Download

Share

COinS