Multi-Tier Design Assessment in the Development of Complex Organizational Systems

Multi-Tier Design Assessment in the Development of Complex Organizational Systems

Melissa A. Dyehouse (Purdue University, USA), John Y. Baek (Center for Advancement of Informal Science Education, USA) and Richard A. Lesh (Indiana University, USA)
DOI: 10.4018/978-1-60566-667-9.ch001
OnDemand PDF Download:


This chapter describes a model for evaluating complex organizations or systems. The design assessment model the authors propose is a response to current notions of assessment. There are assumptions we make about learning and the functioning of complex systems such as academic programs that do not match assumptions that are inherent in traditional forms of assessment. The authors use a case study of Purdue University’s strategic planning process to provide the context for describing how design assessment takes place in a higher education setting. Based on interviews and observations, we identify areas problematic for some notions of assessment and distinguish several implications based on these findings. The design assessment model may be useful when assessing complex educational organizations or programs, such as when (a) educational entities at the university level need to assess new programs or curriculum materials; or (b) curriculum developers need to assess new software or tools for instruction.
Chapter Preview


Assessment and evaluation are increasingly important to evaluate programs or initiatives in higher educational settings. For example, a National Science Foundation (NSF) grant proposal must include an evaluation component (Frechtling, 2002). However, how does one go about evaluating a complex educational entity (i.e., with many interacting/conflicting components, stakeholders, and ideas) in a way that will produce valid and accurate results while balancing time and budgetary constraints?

The kind of multi-tier design assessment procedures discussed in this chapter are based on principles that are well established in “design sciences” such as engineering where: (a) the goals of projects typically involve the development of complex tools, and (b) the underlying design is one of the most important components of the product being assessed. Thus, the development of documentation, as well as knowledge, proceeds in parallel and interactively through sequences of rigorous testing and revising cycles.

The proposed design assessment model is contrasted with other types of assessments that are not appropriate for accomplishing the particular goals of the complex organization under investigation. Purdue University’s strategic planning process uses many of the same methods and has characteristics that are similar to a design assessment model.

This research uses a case study method, drawing upon interviews of key stakeholders involved in a higher educational organization as well as observations that took place during a university strategic planning process for a new Department of Engineering Education. Using this method, we document the main challenges inherent to assessing a complex organization that is attempting to meet the goals outlined in a strategic plan. Next we weave together the process employed by the design researchers in the strategic planning process to the multiple tiers and processes that are used in the design assessment model. The design assessment model, which is based on design research principles, provides a solution to the problems frequently encountered in assessment and evaluation situations in higher education settings.

The main objectives of this chapter are as follows:

  • Identify areas problematic for some notions of assessment (e.g., existing curriculum and assessment standards describe goals for instruction that often do little to clarify how relevant achievements can be assessed, particularly for higher education programs);

  • Use a case study of the strategic planning process as an example of how design assessment practices are employed; and

  • Describe a model of design assessment that is based on design research principles.



We recently published a book about our NSF-funded research on design research principles (Kelly, Lesh, & Baek, in press) and believe that design assessment is the next logical step. Design research is an approach that is becoming more widespread in educational research (Kelly, Lesh, & Baek, in press; Kelly, 2003; Lesh, 2002). Design researchers make use of existing research to develop a product using techniques that engineers commonly employ in cycles of designing, testing, and revising.

Several studies have found positive outcomes about the effects of design research experiences for students’ learning and developmental process, which is summarized by Lesh, Kelly, and Yoon (in press):

…the power and the range of usefulness of their underlying ways of thinking tend to increase significantly. This is because every time they design a new, thought-revealing tool, they are extending and revising the underlying ways of thinking that the tools embody. As a result, the development of the tools involves significant forms of learning, and, as learning occurs, the tools produce auditable trails of documentation that reveal important information about the constructs and conceptual systems that the students are developing. Hence, the activities contribute to both learning and assessment.

Key Terms in this Chapter

Indicators: Measurable states that are used to determine if associated objectives are being met.

Tools: Auditable trails of documentation used to evaluate changes over time.

Design Research: A type of research often taking place in higher education settings with the purpose of developing a product such as curricula or tools by using techniques that engineers commonly employ in cycles of designing, testing, and revising.

Design Assessment: A method of designing the organization/product as it is being measured, which models a system of interacting parts such as professors, administrators, and students who produce documentation that can be used to track changes over time.

Artifacts: Documentation, or a measure of how agents view things.

Evaluation: The use of assessment data to determine whether goals are being met.

Assessment: The process of gathering data in order to make decisions. Assessment is the collection of data while evaluation (see below) uses these data to determine the quality of that which is being assessed and whether or not the goals have been met (Jackman, 2001).

Goals: Broad standards of performance that cannot be measured directly.

Agents: Participants who are part of the organization at all levels and capacities.

Tiers: The levels made up of agents from various parts of the organization.

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Christopher S. Schreiner
Christopher S. Schreiner
Chapter 1
Melissa A. Dyehouse, John Y. Baek, Richard A. Lesh
This chapter describes a model for evaluating complex organizations or systems. The design assessment model the authors propose is a response to... Sample PDF
Multi-Tier Design Assessment in the Development of Complex Organizational Systems
Chapter 2
Hedva Lewittes
In this chapter critical thinking is assessed using two critical thinking learning outcomes that were required for the State University of New... Sample PDF
A Critical Thinking Rubric as the Basis of Assessment and Curriculum
Chapter 3
Suzanne Pieper, Erika Edwards, Brandon Haist, Walter Nolan
The purpose of this chapter is to review literature over the past ten years regarding technology tools that are being used in higher education to... Sample PDF
A Survey of Effective Technologies to Assess Student Learning
Chapter 4
John Baer, Sharon S. McKool
The Consensual Assessment Technique is a powerful tool used by creativity researchers in which panels of expert judges are asked to rate the... Sample PDF
Assessing Creativity Using the Consensual Assessment Technique
Chapter 5
Christine Charyton, Zorana Ivcevic, Jonathan A. Plucker, James C. Kaufman
This chapter discusses creativity assessment as a means for evaluating skills required in higher education. Creativity is assessed in the context of... Sample PDF
Creativity Assessment in Higher Education
Chapter 6
Asao B. Inoue
This chapter articulates writing assessment as a technology, theorized with three aspects (power, parts, and purpose), accounting for the ways in... Sample PDF
The Technology of Writing Assessment and Racial Validity
Chapter 7
Sheila S. Thompson, Annemarie Vaccaro
The purpose of this chapter is to address epistemological and methodological approaches to assessing assessment. The authors’ intent is to show how... Sample PDF
Qualitative and Quantitative Methods as Complementary Assessment Tools
Chapter 8
Teresa Flateby
The development of the Cognitive Level and Quality of Writing Assessment online system is described in this chapter. Beginning with needs identified... Sample PDF
Effects of Assessment Results on a Writing and Thinking Rubric
Chapter 9
Barbara D’Angelo, Barry Maid
Outcomes-based assessment provides data for programs to demonstrate student learning as a result of their enrollment in the program and to assess... Sample PDF
Assessing Outcomes in a Technical Communication Capstone
Chapter 10
Sonya Borton, Alanna Frost, Kate Warrington
As Jacqueline Jones Royster articulated at the 2006 Conference on College Composition and Communication, English departments are already assessing... Sample PDF
Assessing the Composition Program on Our Own Terms
Chapter 11
Joan Aitken
This chapter uses a case study to exemplify one approach to assessment of three instructional delivery formats: (a) online, (b) distance, satellite... Sample PDF
A Case Study of Instructional Delivery Formats
Chapter 12
Victor W. Brunsden
The author present a case-study of a classroom technique that allows assessment and some remediation of several shortcomings of college student... Sample PDF
Inverting the Remedial Mathematics Classroom with Alternative Assessment
Chapter 13
David A. Eubanks
This chapter describes Coker College’s subjective performance assessment program to rate student thinking and communication skills. It uses a... Sample PDF
A Case Study of Authentic Assessment
Chapter 14
P. Tokyo Kang, David Gugin
This chapter reports an outcomes assessment study conducted at the University of Guam. The assessment project was conducted during the 2006-07 and... Sample PDF
Outcomes Assessment in Japanese Language Instruction
Chapter 15
Barika Barboza, Frances Singh
This chapter describes an outcomes assessment study completed in a basic composition course at a small urban open admissions community college. The... Sample PDF
Assessing the Effectiveness of a Basic Writing Course
Chapter 16
Lorraine Gilpin, Yasar Bodur, Kathleen Crawford
Peer assessment holds tremendous potential to positively impact the development of preservice teachers. The purpose of this chapter is to describe... Sample PDF
Peer Assessment for Development of Preservice Teachers
Chapter 17
Charlotte Brammer, Rhonda Parker
In 2002, Samford University began working on a long-term learning assessment plan designed to evaluate its undergraduates’ competencies in written... Sample PDF
Workshops and E-Portfolios as Transformational Assessment
Chapter 18
Daniel F. Chambliss
This chapter describes how the trend favoring assessment initiatives of a system-wide scope such as program review and collegiate learning... Sample PDF
A Neglected Necessity in Liberal Arts Assessment: The Student as the Unit of Analysis
Chapter 19
Deirdre Pettipiece, Timothy Ray, Justin Everett
Perhaps due to its applicability as a performance of skill sets in virtually all disciplines, writing as a mechanism for measuring student success... Sample PDF
Redefining Writing Reality Multi-Modal Writing and Assessment
Chapter 20
Sean A. McKitrick
This chapter introduces methods that can be used to engage faculty in the assessment process, working within a shared governance structure in... Sample PDF
Engaging Faculty as a Strategic Choice in Assessment
Chapter 21
Steven M. Culver, Ray VanDyke
There is much in the assessment literature about the necessity of developing a culture of assessment and mandates from accrediting bodies include... Sample PDF
Developing a Receptive and Faculty-Focused Environment for Assessment
Chapter 22
John Wittman
This chapter argues that as primary stakeholders in writing program assessment, students and instructors need to be included proactively in... Sample PDF
New Collaborations for Writing Program Assessment
Chapter 23
Mya Poe
The study of racial-ethnic group differences on educational tests has yielded a substantial body of research internationally in the last decade. In... Sample PDF
Reporting Race and Ethnicity in International Assessment
Chapter 24
Joan Hawthorne, Tatyana Dumova, April Bradley, Daphne Pederson
In this chapter the authors describe a method developed to assess the outcome of a “cultural familiarity” general education goal. Challenges in... Sample PDF
Method Development for Assessing a Diversity Goal
About the Contributors