Lexical Profiles of Reading Texts in High-Stakes Tests: Where are the Benchmarks?

Lexical Profiles of Reading Texts in High-Stakes Tests: Where are the Benchmarks?

Tan Jin (Sun Yat-sen University, Guangzhou, China), Kai Guo (Northeastern University, Shenyang, China), Barley Mak (The Chinese University of Hong Kong, Hong Kong, China) and Qiuping Wu (Sun Yat-sen University, Guangzhou, China)
DOI: 10.4018/IJCALLT.2017010103
OnDemand PDF Download:
$37.50

Abstract

In language testing literature, the lexical profiles issue has been extensively discussed when examining the quality of reading texts in high-stakes tests. The interpretation and use of lexical profiles, however, have been lacking a point of reference (i.e., benchmarks). Therefore, this study attempts to establish benchmarks for lexical profiles of reading texts in a high-stakes test in China – the National Matriculation English Test (NMET). To elicit sufficient samples, a corpus of 909 NMET reading texts was constructed. Based on the corpus, two stages were employed. Firstly, the 909 texts were screened through a text coverage analysis and representative text samples were selected. Secondly, two sets of benchmarks were established based on the text samples. Overall, this study contributes empirical evidence to evaluating the lexical profiles of NMET reading texts, and has practical implications for developing reading texts in high-stakes tests.
Article Preview

Introduction

A reading comprehension component has been routinely included in high-stakes English proficiency tests for assessing the reading ability of test-takers (Alderson, 2010). In assessing reading ability, a number of texts are employed followed by questions in order to make judgments of test-takers’ levels of reading comprehension, such as IELTS (Taylor & Weir, 2012) and TOEFL iBT (Chapelle, Enright & Jamieson, 2008). In this connection, the preparation of reading texts has been an issue for language testers; trained and experienced item writers are involved to select and edit texts from a large amount of text sources (Green & Hawkey, 2012). To maintain the quality of reading texts, systematic measures are applied in the production process including both qualitative reviews and quantitative analysis (Green & Jay, 2005). In the qualitative reviews of the reading texts, experienced item writers and testing experts are commissioned for pre-editing, editing and pretest review as specified in The IELTS Question Paper Production (IELTS website, 2015). As for the quantitative analysis of the reading texts, one of the major approaches is to detect the lexical profiles (Webb & Paribakht, 2015), which examines the vocabulary proportion of reading texts across vocabulary levels (Coniam, 1999; Nation, 2006). For example, based on a certain word list, 80% of the words in a text may come from the most frequent 1K word families, 10% of the words may come from the second most frequent 1K word families, and the remaining 10% of the words might be at the lower 1K vocabulary levels or even off-list.

While an examination of lexical profiles has been widely used in assessing reading texts in high-stakes tests suited for particular language proficiency levels (Khalifa & Schmitt, 2010), there has been the lack of a point of reference (i.e., benchmarks specifying an acceptable range) to compare and evaluate the values of the vocabulary proportions in the lexical profiles (Coniam & Falvey, 2013). Establishing benchmarks of lexical profiles for evaluating reading texts, however, has been a very challenging issue due to the substantial variation in the lexical profiles of reading texts (Nation, 2006; Webb & Paribakht, 2015). According to Nation (2006), based on an analysis of British National Corpus data, the largest variation in the proportion of vocabulary occurs probably in the most frequent 1K word families, and the range of variation decreases as the word frequency levels decrease. Until recently, it still remained unclear whether lexical profiles of reading texts could be benchmarked with acceptable ranges in the context of high-stakes tests. Furthermore, as claimed by Webb and Paribakht (2015), early studies investigating the issue of lexical profiles in reading tests were conducted using a very limited number of reading texts, which made the setting of benchmarks impossible. Therefore, this study has been designed to fill the gap in exploring the benchmarking issue on lexical profiles by examining a large sample size of reading texts from a corpus of high-stakes test papers in China. Specifically, we operationally define the benchmarks for lexical profiles as a point of reference specifying acceptable ranges for values of vocabulary proportions at certain frequency levels. More importantly, this corpus-informed study is aimed to shed light on the methodology in the research of lexical profiles.

Complete Article List

Search this Journal:
Reset
Open Access Articles: Forthcoming
Volume 7: 4 Issues (2017): 2 Released, 2 Forthcoming
Volume 6: 4 Issues (2016)
Volume 5: 4 Issues (2015)
Volume 4: 4 Issues (2014)
Volume 3: 4 Issues (2013)
Volume 2: 4 Issues (2012)
Volume 1: 4 Issues (2011)
View Complete Journal Contents Listing