Abstract
This chapter gives a relatively non-technical introduction to computer programs for marking of essays, generally known as Automated Essay Scoring (AES) systems. It identifies four stages in the process, which may be distinguished as training, summarising mechanical and structural aspects, describing content, and scoring, and describes how these are carried out in a number of commercially available programs. It considers how the validity of the process may be assessed, and reviews some of the evidence on how successful they are. It also discusses some of the ways in which they may fall down and describes some research investigating this. The chapter concludes with a discussion of possible future developments, and offers a number of searching questions for administrators considering the possibility of introducing AES in their own schools.
TopIi. Automated Essay Scoring Systems
A number of programs (Automated Essay Scoring, or AES, Systems) are described in the literature. These vary in the extent to which they are generally available in a usable form: this chapter will concentrate for the most part on the larger, commercially available systems. The first to be developed, by some margin, is Project Essay Grade (PEG), which first started in the 1960s, but has more recently been revived in an updated form (Page, 2003). Three other major systems are Intellimetric ™ (Elliot, 2003) and the Intelligent Essay Assessor ™ (IEA) (Landauer, Laham and Foltz, 2003) and e-rater ®(Burstein, 2003, Attali and Burstein, 2006). The work of Larkey and Croft (2003) has been significant in this area, and while it appears that there is no corresponding commercially available program, the program BETSY ((Rudner and Liang, 2002)), currently freeware, appears to share much of the same theoretical features.
Key Terms in this Chapter
Corpus: Reference collection text used to establish stylistic or knowledge base for AES.
Latent Semantic Analysis: A technique in natural language processing of analyzing relationships.
Summative Assessment: A form of assessment carried out at the end of a time period and intended to document a learner’s progress.
Automated Essay Scoring (AES): Assigning of a grade to essays using a computer program.
Cross-Validation: Validating a scoring procedure (here, for essays) by applying it to another set of data.
Formative Assessment: A form of assessment intended to give students feedback on their learning progress and to give the teacher an indication of what students have mastered and areas of difficulty.
Singular Value Decomposition: A statistical technique for grouping terms in a document according to meaning.
Natural Language Processing: Use of computers to interpret and manipulate words as part of a language.