Psychometric Post-Examination Analysis in Medical Education Training Programs

Introduction

Assessment in medical education has changed radically over the last two decades (Norcini & McKinley, 2007; Swanwick, Forrest, & O’Brien, 2019). Traditional methods of examinations such as the written papers, based on open-ended questions, and oral examinations are being replaced with standardized written tests of applied knowledge and objective structured clinical exams. These are guided with the intention of making assessment more reliable, valid and acceptable. Recent developments in psychometric theory and access to advanced computational technology have favored such changes, allowing educators to improve methods and tools used to assess students through their academic progression and transition into the medical profession.

The use of psychological measurement in medical education has a long-standing history, originating from the need of educators to make high-stake decisions and provide multiple stakeholders (students, regulatory bodies, health systems, communities) with results in which measurement error is minimized. The vast majority of assessment methods and techniques will ultimately produce indirect observations of latent constructs (Finch & French, 2018), which are assumed to represent part of the fundamental toolkit of knowledge, competences, and skills that a doctor is required to develop (e.g. functional knowledge of basic sciences, diagnostic reasoning, and communication skills). Unfortunately, educational measures differ from the measurement of attributes of physical objects in that they entail a higher degree of variation or error, which in turn need to be inspected and properly addressed to consider the assessment results as reliable and valid, and to allow educators to make confident decisions. This makes the pursuit of precision and the calibration of assessment methods cost or effort dependent, however necessary to satisfy decision-making requirements (Bond & Fox, 2015; Jackson, Jamieson, & Khan, 2007).

Psychometrics represent a set of evidence-based theoretical test models that can help to validate the fundamental assumption that a test score, usually expressed in numeric terms, will represent a certain degree of a medical student’s capacity within a defined domain of scientific knowledge or clinical skills and competencies (Kline, 2014; Norman, 2016). Such models are necessary to clarify and evidence the relation between the measured or observed score, the underlying latent construct that the test is aiming to measure (De Champlain, 2010), and to provide clear and evidence-based recommendations to educators with regards to a process that will determine academic progression and ultimately medical licensing.

It is important to remind the reader that assessment in medical education occurs within a situated system of learning practices and in relation to a curriculum, and the pursuit of a constructive alignment of curriculum and assessment is key in this process (Orsmond & Merry, 2017). The process is meant to be cyclic and guided by the definition and implementation of clear, specific learning outcomes informing all further steps, from the choice of core content to learning, teaching, and assessment methods (Harden, 2007). Assessment will generally serve the purpose of verifying whether medical students have attained – and, to what extent – the desired learning outcomes. Tavakol and Dennick (2011, p. 448) distinguish between two major phases of the assessment process, namely measurement and evaluation. They define measurement as the “process of assigning a numerical value in order to assess the magnitude of the phenomenon being measured”, while drawing upon Tyler's (1949) classic definition, they conceptualize evaluation as “the process of determining to what extent the educational objectives are being realized”. It is at this point that psychometric post-examination analysis comes into place, based on the fundamental aim of evaluating and interpreting the characteristics of a test (e.g. central location and dispersion of scores, internal consistency, construct validity) in relation to the results observed in a group of examinees and the pre-specified learning outcomes of a medical program.