Assessing the effectiveness of academic programs is a critical element of institutional research. Changing demographics in the United States compel us to measure the effectiveness of academic programs to address issues of diversity. At a large public university in the Southwestern U.S., a team of researchers constructed a diversity awareness survey as part of its institutional research. This survey was created in response to several factors: (1) results from a university-wide survey of graduating seniors regarding attitudes toward diversity were alarming; (2) the College of Education was undergoing NCATE accreditation review, and assessment had become part of institutionalized procedures in the college; (3) the Curriculum and Instruction department chair requested it; (4) the teacher certification program was guided by a mission to prepare teachers to be effective in diverse classrooms. This process appeared simple at first, but proved to be a lengthy, complex series of steps on a not-so-linear path. After three years, we are just now reaching a point where we believe the survey validity is strong and the questions need no further revision.
TopBackground
This case study describes the iterative process involved in the development of a diversity awareness pre- and post-survey. This survey, administered to approximately 400 students each semester, is producing a very large amount of data that can be examined in relation to a large number of variables such as age, gender, intended certification level and discipline, and life experiences. Initial findings indicate that most pre-service teachers in the program are at a developmental level of three or four, out of six levels, with regard to the open-ended questions about teaching diverse students. The six developmental levels that we identified for categorizing participants’ awareness/knowledge, skills, and dispositions are:
- 1.
Negative/resistant/no meaningful answer/no answer.
- 2.
Uninformed/superficial understanding/overconfident.
- 3.
Open to learning, but still somewhat uninformed.
- 4.
Beginning to make changes in teaching/develop new ideas/learn from students. Not yet empowering students.
- 5.
Embraces diversity; willing to empower diverse students/culturally responsive.
- 6.
Willing to promote transformation and social change/views teaching diverse students as a privilege.
Quantitative analysis of Likert scale items indicates more openness to diversity than do responses to open-ended questions. Survey improvement is ongoing. The research team is reviewing all survey items to determine if the two types will yield equally valid results or if the Likert scale items allow students to predict an “acceptable” response, while some open-ended questions reveal generally less openness to diversity. Other issues discussed in this chapter include the need for an updated diversity survey, challenges in designing a high-quality survey, identifying emerging categories in quantitative responses, construct validity, developing rubrics for responses to open-ended questions, institutionalizing the survey, continual improvement of survey items, and the protocol for sharing data with colleagues. This survey and other institutional research studies like it are valuable tools for assessing program effectiveness as related to stated university goals, and in making programmatic and curricular decisions as a result.
Reality of Today’s K12 Classrooms
Today’s classrooms call for teachers who are well prepared to instruct diverse students. Unfortunately, classroom teachers often have life experiences that are dissimilar to those of their students. In Texas, over sixty percent of students in public elementary schools in 2005 were non-white (Strayhorn, 2006). A significant number of them speak a first language other than English. In 2006, the U.S. Department of Education estimated that more than 5 million children in U.S. schools, or more than 10 percent, are English Language Learners (National Clearinghouse for English Language Acquisition, 2006). As an example, in the Fairfax, Virginia public schools, at least 41 percent of the students are from homes in which a language other than English is spoken (Dale, 2010).
Some students live in poverty, experience tumultuous home lives, are undernourished, and underrepresented. Some belong to non-mainstream religions and some are GLBTQ (Gay, Lesbian, Bisexual, Transgendered, Questioning). In response to the challenges presented with teaching these students, how does a college of education determine the preparedness of its teacher candidates for these realities? How does a teacher education program assess its effectiveness to cultivate the appropriate awareness, skills, and dispositions to teach such diverse students? This case study traces the work of a research team to develop an instrument for measuring this awareness.