The TPACK framework provides researchers with a robust framework for conducting research on technology integration in authentic environments, i.e., intact classrooms engaged in standards-aligned instruction. Researchers who wish to identify the value added by a promising technology-supported instructional strategy will need to assess student learning outcomes in these environments; unfortunately, collecting valid and reliable data on student learning in classroom research is extremely difficult. To date, few studies using TPACK in K-12 classrooms have included student learning outcomes in their research questions, and researchers are therefore left without models to guide their development, implementation, and analysis of assessments. This chapter draws upon the literature and our own research and assessment experiences in technology-integrated, standards-aligned classroom instruction to give examples and advice to researchers as they develop, analyze, and write up their observations of student learning outcomes. In particular, we focus on standard items, specifically multiple choice items, as an accepted (if limited) method for assessing student understanding. We seek to fill an existing gap in the literature between assessment advice for educational psychologists (who typically work outside of classroom settings) and advice given to teachers (who have lower thresholds for issues such as validity and reliability). Classroom researchers will benefit from this advice to develop, validate, and apply their own objective assessments. We focus on the content areas of science and social studies, but this advice can be applied to others as well.
TopIntroduction
Research on the impact of instructional technology on learning outcomes have a history of finding no significant difference, the technology did not provide a measureable effect on students’ performance (e.g., Clark, 1983; Dynarksi et al., 2007). Kozma (1994) provided a counterpoint, noting that the proper focus is not the technology alone but the technology and the instructional method employed by the teacher. Mishra and Koehler’s framework of Technological Pedagogical Content Knowledge (TPACK; Mishra & Koehler, 2006) provides an even more comprehensive set of variables as it broadens Kozma’s interest in instructional method into the richer, more descriptive context of Shulman’s Pedagogical Content Knowledge (1987). Researchers can use TPACK to frame studies that attend to the interacting variables of content, pedagogy, and technology in the authentic environment of technology integration: classroom teachers engaged in standards-aligned instruction with intact groups of primary and secondary students. These more fully-contextualized, real-world studies may be able to shed light on which combinations offer no relative advantage and which do, indeed, afford a significant difference in learning outcomes.
As Schrum et al. (2007) noted in their discussion of TPACK, “Until the pedagogical methods that uniquely take advantage of a technology’s pedagogical affordances to achieve content-specific learning objectives are identified, it will not be possible to prepare teachers to make effective use of current and emerging technologies” (p. 460 [emphasis added]).The goal of this chapter is to review the state of the field in TPACK-related research to see whether and how this task is being carried out and to offer constructive, specific guidance to future research. Specifically, we will
- 1.
Examine the extent to which TPACK-informed research has sought to observe the relative advantage of technology integration strategies in terms of student learning in elementary and secondary classrooms,
- 2.
Identify and evaluate the methodology used in this research to identify exemplars, and
- 3.
Advise TPACK-informed researchers as they conduct their own assessments of student learning in the authentic environment of technology integration: intact primary and secondary classrooms engaged in standards-aligned instruction.
In the spirit of TPACK’s attention to context, we will focus on the content areas of science and social studies, these are the areas of our own instructional and research expertise and are therefore the ones in which we are most competent to review others’ work with a critical but constructive lens. Our analysis and advice can inform work in other content areas as well but will directly address only these designated areas.
TopAs an initial sampling frame of TPACK-informed literature, we selected the TPACK Reference Library hosted by Koehler and Mishra (2010). The Reference Library contains more than 150 citations covering theory, pre-service teacher education, professional development for in-service teachers, and classroom practices of technology integration. The list is self-maintained by members of TPACK community. The most recent update (as of our analysis) was January 15, 2010. Of the 150 citations provided, fewer than 10 present intact classroom studies. Of these 10, only 3 include student academic performance and/or learning outcomes as one of their targeted variables (Doering & Veletsianos, 2007; Manfra & Hammond, 2008; Warren, 2006). All three studies are qualitative; the researchers did not measure student learning but only described behaviors consistent with academic goals. These goals included meeting language arts and technology standards (Warren, 2006), geography education objectives (Doering & Veletsianos, 2007), and concordance with the classroom teachers’ standards-aligned history instruction (Manfra & Hammond, 2008). While these studies can inform future research and practice, they do not provide models for researchers who wish to use TPACK to identify instructional approaches that improve students’ learning outcomes relative to other strategies (cf Schrum et al., 2007). Our conclusion from this initial literature review of the Reference Library is that the TPACK community and literature base is currently under-served in terms of models for assessing student learning.