Abstract
Innovative item formats are attractive to the sponsors of professional certification or qualification examinations because they provide greater fidelity to the real world than traditional item formats. Using the design of the Chartered Institute of Management Accountant's professional qualification examinations as a case study, this chapter presents an in-depth exploration of the issues surrounding the use of innovative items to assess higher-order thinking skills required for professional competency, beginning with a discussion of approaches taken by various academic disciplines to define and characterize higher order thinking. The use of innovative, authentic assessments is examined in the context of validity arguments. A framework for principled thinking about the construct map of the assessment is introduced, and a systematic process for designing innovative items to address the desired constructs is provided.
TopBackground
Based in the United Kingdom, CIMA is the world's largest professional body for management accountants. CIMA sponsors an entry-level Certificate in Business Accounting and three levels of professional qualifications that lead to a candidate's becoming a chartered management accountant. Annually, approximately 60,000 candidates across the world sit for CIMA's professional qualifications.
Project 2015 was a comprehensive process undertaken to revise CIMA's professional qualifications learning curricula. An accompanying redesign of its certification examinations took advantage of the capabilities of CBT. CIMA's Project 2015 aimed at ensuring that the professional learning and examinations programs remained relevant and reflective of the real world of business. Figure 1 relates the high-level process used to define the learning and assessment products that resulted from Project 2015.
Figure 1. CIMA’s design process and resulting learning and assessment process
Competency Framework
The first step in the Project 2015 process for redesigning the professional qualifications was the development of a Competency Framework. To develop the Framework, comprehensive global research was conducted with finance professionals and employers of various sizes and in different sectors. This research included a written questionnaire sent to CIMA members and students across the globe, supplemented by face-to-face meetings and roundtable discussions with relevant organizations and stakeholders.
The resulting Competency Framework outlined the knowledge, skills, and abilities management accountants need to maintain employment and to deliver sustainable success for their organizations. The Competency Framework consisted of four types of skills (technical, business, people, and leadership) employed by management accountants at three practice levels (operational, management, and strategic). The Framework recognized that the emphasis on the particular skills varies throughout an accounting career. As reflected in Figure 2, entry-level roles at the operational level of practice require greater emphasis on technical accounting skills; in contrast, more senior roles at the strategic level place more emphasis on business acumen and on people and leadership skills.
Figure 2. Emphasis of competencies by level of professional practice
Key Terms in this Chapter
Scalability: The ability to develop test content by applying a single item template to different scenarios and situations in order to develop a multitude of test items.
Item Responsibilities: The facets of a test item that must be considered when designing computer-based assessments. These facets relate to the user interface and presentation of the item, how the item is selected and ordered within the assessment, the context within which the item is delivered, how the test taker navigates to and away from the test item, and how the item response is assessed and scored.
Configurability: Design of individual test items (or a group of test items as a collective) which has a specific format and composition that can be re-used as a pattern for new test content.
Principled Item Development Methodology: Any one of a number of methodologies which provides a systematic process for developing test content with the aim of refining the construct responsibilities of the item so that developmental efforts can be better targeted with regard to the measurement that test items will ultimately provide.
Case Study: A small collection of test items that are all related to a single scenario; the scenario itself may contain any number of stimuli related to the test items. These stimuli may be graphic, such as charts or diagrams, or may be text-based, such as reports or tables.
Authentic Assessment: A test that has greater fidelity to how tasks would be performed in the real world.
Construct Definition: The process of determining the traits (such as knowledge, skills or abilities) that are desired to be measured in a test or assessment.
Higher-Order Thinking Skills: The complex mental processes required to evaluate, analyze, and synthesize information in order to make decisions.
Innovative Item: A test item that is formatted in a way that is not found in traditional pencil-and-paper assessments.