The researchers included 10 embedded assessments in the study for a mean gain effect size of .35. From the results of a pilot study, the authors learned that while the technique works, questions about the reliability and validity of the classroom assessments emerged. The authors speculate that institutional researchers can use these techniques to compare the assessment of standards within and among universities.
TopBackground
Increasingly, regional accrediting groups and state accrediting bodies require improved assessments of student learning. The question of how best to produce evidence of student learning is an increasingly important one, specifically since these external groups have uniformly agreed that courses grades are not enough. In fact, in the last decade, an industry has emerged that serves this purpose. One classic external assessments include the National Survey of Student Engagement (NSSE). This assessment focuses on student motivation to engage both curricular and extracurricular activities on campus. Another large assessment, called the Collegiate Learning Assessment (CLA), focuses more on critical and analytical reasoning and implied and explicit outcomes of students’ encountering the college curriculum. Once the university collects these data, it can get a broad profile of student learning. However, both the NSSE and the CLA (as examples of a class of university-wide assessments) fall short because they are not necessarily aligned with what actually occurs in a college classroom, which limits their usefulness to curriculum change. Therefore, even though these measures have demonstrated high favor among those who evaluate programs, the instruments may not be valid for a particular institution that has learning objectives that are specific that institution.
Meta-analysis is a statistical technique designed for summarizing the results of quantitative studies. Lipsey and Wilson (2001) call meta-analysis a form of survey research in which researchers survey studies rather than respondents. Meta-analysis is particularly useful when aggregating findings across a broad field of study, even if the studies use different summary statistics. In his original meta-analysis, Glass (1976) combined the results of different psychotherapies drawn from the entire field of psychology. He was able to draw powerful conclusions about the overall effectiveness of psychotherapy and insightful comparisons about the differences among various approaches, despite the use of different measurement tools. In an analogous fashion, meta-analysis can be used in program evaluation when a broad group of diverse courses share the same student learning objectives. This can be a powerful assessment design that complements external standardized external assessments with instructor-generated embedded assessments.
As a response to the limitations of external assessments, many institutions are turning to embedded assessments, where assessment resides with the instructor and data gathering and reporting is flexible enough to allow participation from disparate fields (Gerretson & Golson, 2005). These assessment activities mimic typical course activities, so they are invisible to students. However, instructional and assessment approaches among instructors will certainly vary, even among instructors teaching the same course. This forces assessment specialists to find innovative ways of determining student proficiency on a learning objective across many different types of embedded assessments. This chapter proposes that meta-analysis may be the most organized and systematic approach to harnessing the power of instructor-driven embedded assessments.
Traditionally, institutions draw upon standardized external assessments to help provide evidence of student learning in the general education curriculum. These assessments can be expensive. Often the external assessments do not exactly match student learning objectives Additionally, institutions may resort to unusual steps in order to create a sample of needed respondents. In fact, when and how samples are created can confound results. For example, just before the due date for an accreditation self-report, the researchers have personally seen institutional research and planning officers trolling the student union building handing out t-shirts or basketball tickets to any undergraduate who will take out two hours to complete a battery of assessment tools. Convenience sampling techniques are non-trivial and can lead to invalid results because the resulting samples are rarely representative.