Abstract
Writing remains a staple of academic evaluation instruments, and for good reason. Few other tools come close to writing in assessment of the student's ability to formulate coherent arguments, demonstrate critical thinking, and present explanations. Conscientious instructors mark-up and comment on student writing, evaluating its success in meeting content goals, structure, relevance of evidence, grammar, mechanics, and style. It is an exhausting and time-consuming process but remains the single best way to support developing writers and thinkers. Technology to detect patterns of errors and to offer feedback has evolved over time, but can the programs do what professors do? This chapter provides a status report on automated writing evaluation and its role in higher education. A balance can be struck between the efficiency of tech tools and professor judgment. Recommendations for automating and expediting the review of student writing are offered, with a focus on remote learning environments.
Author’s Note: Numerous phrases for the same concept of automated writing review have emerged over time. Automated Essay Scoring, Automated Essay Evaluation, Automated Writing Evaluation, Automated Essay Grading, Automated Essay Assessment, and Automated Essay Scoring are all synonymous. Automated writing evaluation (AWE) is the preferred term in this chapter as it encompasses all writing rather than the limited genre of the essay form.
TopAutomated Writing Evaluation Background
Research supports the role of automated writing evaluation as a practical solution to the onerous task of manually grading student writing. Despite this potential value, over the past half century, many products have emerged and disappeared, defeated by the many challenges of determining validity and reliability, evaluating semantic content, and providing useful feedback (Zupanc & Bosnic, 2015). Writing assignments that best fit the automated review system are prompt-specific, meaning all are on the same topic. Human assessors score the writing to construct a learning set that is then used to develop the scoring model of the automated evaluation. The model is then applied to score new, ungraded writing. The performance of the model is validated by determining how well it replicated the scores assigned by human graders (Parra & Calero, 2019; Zeide, 2019). As the pool of writing samples increases, scoring efficiencies also, theoretically, become more accurate and thereby more useful. Inherently problematic in application of automated writing evaluation is the prompt-specific requirement. Student writing remains the single best way to assess student learning because of its demands on presentation of complex functions. Unlike objective questions, good writing assessments demand the ability to recall, integrate, organize, and present, all challenging actions. The measure of these outcomes, corresponding to the higher levels of Bloom’s (1956) taxonomy - evaluation and synthesis – brings the greatest value to the student written response. Prompt-specific, defined answer questions do not provide the depth and value of complex writing assignments (Yu, 2019).
Key Terms in this Chapter
Writer Self-Efficacy: Psychologist Albert Bandura proposed the concept of self-efficacy as an individual’s ability to assess their own capacity to execute a certain course of action. Writer self-efficacy is the writer’s personal judgment of their own ability to write effectively.
Feedback: Occurs when outputs in a system are routed back to the original source with evaluative observations to improve the output.
Text Expanders: Custom abbreviations that allow users to insert words or phrases automatically with a few simple keystrokes or abbreviations.
Transactional Distance: Theory developed by Michael Moore in the 1970s to consider teaching and learning conducted through technology rather than theories developed for classroom application.
Critical Thinking: The analysis of facts to arrive at a judgment or conclusion. Involves consideration of rational, unbiased evaluation of evidence.
Recursive Nature of Writing and Learning: Writing and learning are not linear processes but rather involve back and forth movement to return to earlier steps, revise, and reconsider.
Automated Assessment Tools: Software designed to replicate grader review of student writing for purposes of standardized grading reviews and reduce onus on human graders.
Macros: Short for “macroinstruction” macros are used to present a sequence of instruction as a single program statement to reduce tedium and errors.
Artificial Intelligence: Intelligence demonstrated by machines, as opposed to natural intelligence inherent in humans and animals that includes both consciousness and emotionality.
Assessment of Writing: Refers to an area of study that includes theories and practices used to guide the evaluation of a writer’s performance.