Abstract
The purpose of this chapter is to provide an overview of prevalent measures for evaluating the quality of system output in seven key text mining task domains. For each task domain, a selection of widely used, well applicable measures is presented, and their strengths and weaknesses are discussed. Performance evaluation is essential for text mining system development and comparison, but the selection of a suitable performance evaluation measure is not a straightforward task. Therefore this chapter also attempts to give guidelines for measure selection. As measures are under constant development in many task domains and it is important to take the task domain characteristics and conventions into account, references to relevant performance evaluation events and literature are provided.
TopBackground
The evaluation of text mining systems can be broadly divided with respect to its goals into performance evaluation, adequacy evaluation and diagnostic evaluation (Hirschman & Thompson, 1997). Performance evaluation, the focus in this chapter, measures the system performance in one or more specific areas, typically producing numeric values. By contrast, adequacy evaluation aims at establishing to what extent the system satisfies user needs in a particular task and diagnostic evaluation determines the correctness of the system output, typically with respect to a test suite. These latter aspects of evaluation are not considered in more detail in this chapter; for further information, see, for example, Hirschman and Thompson (1997).
Performance evaluation contains three levels of specificity, namely the criterion level, measure level and method level (Spärck Jones, & Galliers, 1996, pp. 19-20; Hirschman & Thompson, 1997). At the criterion level, the aspect of system performance to be measured is selected. Broadly, the criteria can be divided into two separate types, intrinsic criteria are those assessing the performance of a text mining system component as an isolated unit unconnected to the other system components, and extrinsic criteria are those determining its effects on the overall performance of the system. At the measure level, a suitable performance evaluation measure is chosen to reflect the criterion of interest. Finally, at the method level, the method to determine the appropriate value for the given measure and system is designed.
Key Terms in this Chapter
External Measures: For clustering evaluation are based on comparing the clustering system output against a gold standard clustering. Measures such as Rand Index, Jaccard coefficient and F measure can be used as external measures. See also internal measures.
Content Similarity Measure: At the core of text summarization evaluation is the measuring of content similarity between two summaries. The content similarity measure can be lexical (based on word or sentence units, e.g. cosine similarity measures) or semantic (based on semantic content units, e.g. Summarization Content Units in the pyramid method).
Rank Correlation Coefficient: Is a measure of the association between two rankings produced from the same instances. Examples of rank correlation coefficients are Spearman’s ?b and Kendall’s tb.
Internal Measures: For clustering evaluation are based on measuring the extent to which the obtained clustering matches the information inherent in the data. Measures such as Dunn index and Davies-Bouldin index as well as correlation coefficients like Hubert’s G can be used as internal measures.
Gold Standard: Is a dataset defining the correct outputs, often determined by human experts. The output of the evaluated system is compared against the gold standard in order to measure the performance.
Performance Evaluation Measure: Is a real-value function assessing the quality of the text mining system output. The measure could be, for example, the number of fully correct outputs or the number of errors per input instance.
Intrinsic Evaluation: Assesses the performance of a text mining system component as an isolated unit unconnected to the other system components. See also extrinsic evaluation.
2 × 2 Contingency Matrix: Is a 2 × 2 matrix whose columns and rows represent the output of the system and the gold standard, respectively. The elements of the matrix are the numbers of true positives, false negatives, false positives and true negatives, upon which performance measures such as accuracy, precision and recall, are based.
Extrinsic Evaluation: Assesses the performance of a text mining system component from the perspective of its effects to the performance of the whole system. See also intrinsic evaluation.