This chapter reports the results of a pilot study investigating the potential of ChatGPT to assess (or grade) texts produced by international graduate students whose mother tongue is other than English (IGS). The chapter overviews some suggested applications of ChatGPT to academic writing (AW) and particularly, for English as an additional language (EAL) contexts. Most works up to date consider ChatGPT applications for literacy and writing skills development. The authors compare evaluations of texts by ChatGPT vis-à-vis human experts. The texts employed for the study are short literature reviews written by IGS in Canadian universities. The results demonstrate some potential of ChatGPT for assessment. However, while ChatGPT provides mostly relevant scores and comments related to the assessment rubrics provided, the comments are not diversified and individualized enough. ChatGPT queries for assessment should be modified and the tool trained in diversifying comments to achieve better results.
TopIntroduction
ChatGPT has been called “the best artificial intelligence chatbot ever released to the general public” (Roose, 2022, nPg). It is the fastest-growing technology application in history (and caused an avalanche of discussions and research studies Ciampa et al., 2023, p. 186). A considerable amount of debate surrounding ChatGPT revolves around its use in academic writing (e.g., Guo et al, 2023). The proponents of ChatGPT note its multiple applications in diverse areas, such as in creating dictionaries (Lew, 2023), in healthcare (Salam, 2023), in medical education (Lee, 2024), in producing research papers from scratch (Conroy, 2023) supporting the writing of medical articles (Benichou & ChatGPT, 2023), etc. Anecdotally, some individuals even tried to make ChatGPT write new recipes, but it did not go so well (Roose, 2022). Some scholars point out ChatGPT’s potentially negative effect on society including fake news, false information, unlawful use, stereotyping, plagiarism, social security breaches, reduced human interaction, ethical concerns about privacy, and the potential for bias (Ciampa et al., 2023; Dempere et al., 2023; Guo et al, 2023). The fears of ChatGPT have even caused some countries, such as Italy, to temporarily ban its use (Shidarta & Martinelli, 2023). To alleviate potential negative effects, ‘responsible’ use of ChatGPT has been suggested (e.g., Yang, 2023) along with training students how to use it ethically and responsibly (e.g., Ciampa et al, 2023).
Multiple earlier studies have provided accounts of ChatGPT's role in developing literacy and writing skills (e.g., Ciampa et al., 2023; Creely et al., 2023). Writing skills development is a concern shared by many instructors in North American universities especially when it comes to writing by international students who are non-native speakers of English (e.g., Okuda & Anderson, 2018; Li et al., 2023). The authors of this article have also been concerned with the quality of academic writing produced by international graduate students whose mother tongue is other than English (IGS) in their graduate programs of studies. Therefore, the focus of this paper is graduate level English as an Additional language (EAL) writing. For simplicity, we understand EAL here as being composed of English as a Foreign (EFL) or as a Second (ESL) language (the differences between the terms that we ignore here are described in Nayar, 2021). EAL writers are known to experience “the triple disadvantage of having to read, do research and write in another language” (Van Dijk, 1994, p. 276).
In the study reported in this chapter we address the capability of ChatGPT to provide adequate evaluations of EAL students’ writing at a graduate level (international graduate students’ short Literature Reviews). The writing samples were produced by IGS participants in an online free open Literature Review Writing (LRW) Tutorial developed by the authors (Authors 2022, 2023). The idea of the Tutorial came in response to the local Canadian context and was inspired by Genre analysis (Halliday, 1978; Halliday & Matthiessen, 2014) and Computer Assisted Language Learning (CALL) approaches (Hegelheimer & Lee, 2013; Yeh, 2015) to writing skills development.
In Canada, not many universities have adequate resources for providing Academic writing courses to IGS in general and in their disciplines in particular. In an attempt to remedy the above issues, the authors developed an extracurricular free Academic Writing Tutorials set for graduate students in Linguistics and Social Sciences with a focus on the genre of Literature Review. The choice of the genre was determined by its high frequency among the different types of assignments at graduate level in Canadian universities across different programs of studies (Shi & Dong, 2015). The purpose of the tutorial was to provide additional online Academic Writing support resources for IGS to enhance their academic writing skills and to release the EFL/ESL grading problem off the instructors through cost-efficient e-learning. The writing samples analyzed in our paper come from this online tutorial.