This chapter provides a case study describing the process of using theory in practice in implementing higher level evaluation of training initiatives within one unit of a global financial services organization. The organization has used Kirkpatrick's level one and two evaluations for many years to evaluate training. This case describes the first time a level three evaluation was used, covering why it was necessary, the process for developing, as well as the outcomes of the evaluation. This chapter also describes the influence of organizational culture in the decisions made in the implementation of the evaluation strategy.
TopIntroduction
Imagine a fast paced business environment with multiple initiatives rolling out every month. The pressure to perform is fierce. The demand to deliver is unrelenting. Sound familiar? That’s because it describes so many organizations, especially financial institutions that survived the recession of 2008. If training, whether classroom or on-line, is the solution chosen to support business results, how can an organization be certain the training offered meets the business’ need to provide the company a competitive advantage to survive? One way to understand the impact of learning initiatives is to effectively evaluate them (Caffarella & Daffron, 2013; Kirkpatrick & Kirkpatrick, 2007, 2005; Philips & Stone, 2000; Pineda, 2010; Russ-Eft &Preskill, 2009; Short, 2009).
When asked if their organizations evaluated training, all five of the Human Resource and Training managers questioned responded with an unequivocal “yes” or, “of course.” When asked how they measured transfer of learning or the impact of the training on the organization, the responses were not as forthcoming. Most of the managers described their evaluations as end of training “reaction-type” evaluations (Kirkpatrick & Kirkpatrick, 2007; Phillips & Stone, 2000; Russ-Eft & Preskill, 2009), asking how the instructor accomplished the objectives of the program, how prepared the instructor was for conducting the training, and if the information would be relevant to their jobs. We all understand this type of evaluation, and its importance in the training process. Learners are customers of the training department and need to feel that their comments are valued (Kirkpatrick & Kirkpatrick, 2007). Trainers can also use these end of training reaction evaluations to focus the program for other groups. Other managers described “pre-test and post-test” as part of their evaluation process (Cafarella & Daffron, 2013; Kirkpatrick & Kirkpatrick, 2009; Russ-Eft & Preskill, 2009). While this is a step towards understanding learning from the training experience, in practice, these types of evaluations do not provide the organization the information to understand if the training provided will impact their bottom line. “Being ‘satisfied’ doesn’t necessarily mean that they learned anything” (Kirkpatrick & Kirkpatrick, 2007, p. 47).
While most organizations would claim to want an evaluation of training to understand the impact on business metrics, it is the organizational culture and the readiness of management to devote the time and resources to this effort that ensures that higher–level evaluations of training take place (Lewis & Thornhill, 1994; Russ-Eft & Preskill, 2009). Organizations in the throes of trying to beat the competition and stay “on top” may want to understand how their training is impacting the organization, but will they commit the time and resources to gather this information? While they want to understand the return on the investment of the training, are managers knowledgeable and willing to perform the steps needed to follow up with trainees and their supervisors to gather this information?