Abstract
There are different methods to examine the predictive validity of an instrument. In this chapter, the author presents a method of validation—cross sample analysis, using a study as an example. This study demonstrates the procedures to determine whether a technology attitude instrument can predict student technology learning achievement consistently across four featured samples, with the data from two universities over a nine-year period. A base-model of prediction is first developed and then tested. The predictive validity of the instrument is confirmed by the model testing results that no significant differences exist between the means of the predicted and observed learning achievement scores in each featured sample group. Background knowledge and other relevant methods of validation are also reviewed in this chapter.
TopIntroduction
In the field of using information technology in education, assessment has been a weak area (Liu & Handerson, 2003; Liu & Maddux, 2008). Literature shows that lack of validation of instruments used in the field is one issue related to this weakness (Maddux & Cummings, 1999). Educators and researchers often use self-developed instruments to assess student performance, instructional design, learning outcomes, or the effectiveness of technology integration (Christensen & Knezek, 2001). Unfortunately, in many studies conducted with such self developed instruments, there are no reports about the validity of the instrument (Liu & Maddux, 2008). If the validity of the instruments is not confirmed, that is, if they are not measuring what they are supposed to measure, the results of the studies cannot be considered meaningful; and such studies would not provide any valuable insights to the field, to the literature, or to the practice of other educators.
There are multiple forms of validity, depending on the purpose of the instrument, research questions, and specific type of inference the study intends to make. Procedures and statistics methods of validation are varied, and thorough validation of any instrument is a complex task that requires careful design and may take years to complete (Bryant, 2004; Moody, 2001; Strube, 2004). This chapter will focus on predictive validity and an applied method of validation—cross sample analysis. The cross-sample analysis is performed in a study that examines whether a technology attitude instrument measuring four attitude variables can predict student technology learning achievement consistently across four featured sample groups, with the data from two universities over a nine-year period.
Key Terms in this Chapter
Predictor Variable: A variable used in regression to predict another variable. It is sometimes referred to as an independent variable if it is manipulated rather than just measured.
Validation: The process to determine the degree of validity of a measure.
Cross-Sample Analysis: A method to test a model with multiple featured samples.
Model Testing: Using statistics procedures to examine and confirm a model, usually an initial model is developed first, and then tested and examined.
Criterion Variable: The variable being predicted in regression. It is the dependent variable.
Over-Time Analysis: A method to test a model following one sample or multiple samples over time.
Technology Attitude: The extent to which one person feels about using and learning technology.
Predictive Validity: The degree to which an instrument can predict what it is supposed to predict.