Hey, ChatGPT, Look at My Work: Using Conversational AI in Requirements Engineering Education

The emergence of conversational AI tools in late 2022 practically changed the face of software engineering and software engineering education. Contemplating the question of how to best prepare and evaluate students in this new reality, we experimented with systematically introducing a conversational AI tool, ChatGPT, into the 2023 offering of an upper-level undergraduate project-based course on Software Engineering. In this course, 20 groups of four students each had to design and implement a project of their choice, with an Android-based mobile client and a Node.js-based cloud server. This paper discusses our goals, approach, and lessons learned from introducing ChatGPT into the first phase of the project development: scoping the work and defining the project requirements.

Students were allowed to use ChatGPT at any stage of their project, albeit in a controlled way, to fulfill the educational objectives of the course. Our experience shows that students can achieve comparable results using a variety of ChatGPT interaction modes and the success of each mode largely depends on students' preferences, learning styles, and the invested effort. Yet, in any of the modes, with moderate effort, students can produce artifacts of a mid-range quality level of around 80%. Moving above this range requires substantial investment, which can be spent on brainstorming, crafting high-quality prompts, or critically assessing ChatGPT's output. We also observe low prompting proficiency of the students: students can improve their prompting strategies by providing a more adequate description of their course and project setup, examples, and expected output format for their requests. Interestingly, students can often be "swayed" by ChatGPT's projected confidence, even when their original ideas are, in fact, more appropriate than the proposed refinements.

• Software and its engineering → Requirements analysis; • Applied computing → Education.