Lexical Profiles of Reading Texts in High-Stakes Tests: Where are the Benchmarks?

Introduction

A reading comprehension component has been routinely included in high-stakes English proficiency tests for assessing the reading ability of test-takers (Alderson, 2010). In assessing reading ability, a number of texts are employed followed by questions in order to make judgments of test-takers’ levels of reading comprehension, such as IELTS (Taylor & Weir, 2012) and TOEFL iBT (Chapelle, Enright & Jamieson, 2008). In this connection, the preparation of reading texts has been an issue for language testers; trained and experienced item writers are involved to select and edit texts from a large amount of text sources (Green & Hawkey, 2012). To maintain the quality of reading texts, systematic measures are applied in the production process including both qualitative reviews and quantitative analysis (Green & Jay, 2005). In the qualitative reviews of the reading texts, experienced item writers and testing experts are commissioned for pre-editing, editing and pretest review as specified in The IELTS Question Paper Production (IELTS website, 2015). As for the quantitative analysis of the reading texts, one of the major approaches is to detect the lexical profiles (Webb & Paribakht, 2015), which examines the vocabulary proportion of reading texts across vocabulary levels (Coniam, 1999; Nation, 2006). For example, based on a certain word list, 80% of the words in a text may come from the most frequent 1K word families, 10% of the words may come from the second most frequent 1K word families, and the remaining 10% of the words might be at the lower 1K vocabulary levels or even off-list.

While an examination of lexical profiles has been widely used in assessing reading texts in high-stakes tests suited for particular language proficiency levels (Khalifa & Schmitt, 2010), there has been the lack of a point of reference (i.e., benchmarks specifying an acceptable range) to compare and evaluate the values of the vocabulary proportions in the lexical profiles (Coniam & Falvey, 2013). Establishing benchmarks of lexical profiles for evaluating reading texts, however, has been a very challenging issue due to the substantial variation in the lexical profiles of reading texts (Nation, 2006; Webb & Paribakht, 2015). According to Nation (2006), based on an analysis of British National Corpus data, the largest variation in the proportion of vocabulary occurs probably in the most frequent 1K word families, and the range of variation decreases as the word frequency levels decrease. Until recently, it still remained unclear whether lexical profiles of reading texts could be benchmarked with acceptable ranges in the context of high-stakes tests. Furthermore, as claimed by Webb and Paribakht (2015), early studies investigating the issue of lexical profiles in reading tests were conducted using a very limited number of reading texts, which made the setting of benchmarks impossible. Therefore, this study has been designed to fill the gap in exploring the benchmarking issue on lexical profiles by examining a large sample size of reading texts from a corpus of high-stakes test papers in China. Specifically, we operationally define the benchmarks for lexical profiles as a point of reference specifying acceptable ranges for values of vocabulary proportions at certain frequency levels. More importantly, this corpus-informed study is aimed to shed light on the methodology in the research of lexical profiles.