This chapter describes a model for evaluating complex organizations or systems. The design assessment model the authors propose is a response to current notions of assessment. There are assumptions we make about learning and the functioning of complex systems such as academic programs that do not match assumptions that are inherent in traditional forms of assessment. The authors use a case study of Purdue University’s strategic planning process to provide the context for describing how design assessment takes place in a higher education setting. Based on interviews and observations, we identify areas problematic for some notions of assessment and distinguish several implications based on these findings. The design assessment model may be useful when assessing complex educational organizations or programs, such as when (a) educational entities at the university level need to assess new programs or curriculum materials; or (b) curriculum developers need to assess new software or tools for instruction.
TopIntroduction
Assessment and evaluation are increasingly important to evaluate programs or initiatives in higher educational settings. For example, a National Science Foundation (NSF) grant proposal must include an evaluation component (Frechtling, 2002). However, how does one go about evaluating a complex educational entity (i.e., with many interacting/conflicting components, stakeholders, and ideas) in a way that will produce valid and accurate results while balancing time and budgetary constraints?
The kind of multi-tier design assessment procedures discussed in this chapter are based on principles that are well established in “design sciences” such as engineering where: (a) the goals of projects typically involve the development of complex tools, and (b) the underlying design is one of the most important components of the product being assessed. Thus, the development of documentation, as well as knowledge, proceeds in parallel and interactively through sequences of rigorous testing and revising cycles.
The proposed design assessment model is contrasted with other types of assessments that are not appropriate for accomplishing the particular goals of the complex organization under investigation. Purdue University’s strategic planning process uses many of the same methods and has characteristics that are similar to a design assessment model.
This research uses a case study method, drawing upon interviews of key stakeholders involved in a higher educational organization as well as observations that took place during a university strategic planning process for a new Department of Engineering Education. Using this method, we document the main challenges inherent to assessing a complex organization that is attempting to meet the goals outlined in a strategic plan. Next we weave together the process employed by the design researchers in the strategic planning process to the multiple tiers and processes that are used in the design assessment model. The design assessment model, which is based on design research principles, provides a solution to the problems frequently encountered in assessment and evaluation situations in higher education settings.
The main objectives of this chapter are as follows:
- •
Identify areas problematic for some notions of assessment (e.g., existing curriculum and assessment standards describe goals for instruction that often do little to clarify how relevant achievements can be assessed, particularly for higher education programs);
- •
Use a case study of the strategic planning process as an example of how design assessment practices are employed; and
- •
Describe a model of design assessment that is based on design research principles.
TopBackground
We recently published a book about our NSF-funded research on design research principles (Kelly, Lesh, & Baek, in press) and believe that design assessment is the next logical step. Design research is an approach that is becoming more widespread in educational research (Kelly, Lesh, & Baek, in press; Kelly, 2003; Lesh, 2002). Design researchers make use of existing research to develop a product using techniques that engineers commonly employ in cycles of designing, testing, and revising.