The purpose of this chapter is to describe a successful case study of evaluation and learning transfer in a Global CRO (ICON) located in Dublin, Ireland. The case study focuses on a management development program entitled Management Essentials Two. The evaluation element of the exercise engaged the theories of Kirkpatrick and Phillips as a fundamental approach. The outcomes of the program that were measured included Levels one, two and three of both models. Post program behavior change was selected as a measure of transfer. In a departure from standard evaluation practice the case study employed an original inventory, “System of Training Transfer Assessment Report” (STAR), to measure the factors leading to transfer (Donovan & Darcy, 2011). Regression analysis was used to identify four factors that were critical to transfer of learning at ICON. Lessons for ICON, practitioners, and researchers are drawn.
TopIntroduction
This case study concerns a comprehensive evaluation and transfer encounter within ICON who are a global provider of outsourced development services to the pharmaceutical, biotechnology and medical device industries. Originating in Ireland, ICON operates 77 offices in 38 countries across three continents, has 11,700 employees and has its headquarters in Dublin, Ireland. The training event under consideration was conducted throughout the global organization and involved over 600 participants. The evaluation event was executed with the assistance of an external consultant team of which the author was a principal member. The overall purpose of the evaluation event was not only to identify the changes at the behavioral level following the training but also to assess the transfer system operating in ICON and to specify which of the factors leading to transfer were critical in affecting the resultant change in behavior.
The training event was a management development program entitled Management Essentials 2. Management Essentials 2 is a five-day behaviour change classroom-based program designed to build on introductory management training but which is also is seen as a good entry point for seasoned managers at ICON who don’t necessarily have direct reports. The program is delivered in two modular phases. Module A (3 days) deals with the ICON context, personal effectiveness, managing work effectively, managing performance, and coaching conversations. Module B (2 days) is concerned with team management, communications, and personal resilience. An application phase of 4 to 6 weeks was inserted in between Modules A and B to allow for practice of the learning.
Management Essentials 2 was selected as a case study because of its strategic importance to ICON and because of its scale (600 trainees). It required considerable investment on the part of management in terms of time and other resources. Because it was well designed as a learning event it also lent itself to the Kirkpatrick and Phillips models of evaluation. In addition, the management of ICON committed to executing a comprehensive evaluation and transfer assessment that would provide significant insight into optimal ways of deploying training initiatives for the future thus offering interesting research opportunities.
The evaluation and transfer design was a pre and posttest behavioral measure scheme using manager assessment of the behavior of program participants. The measures were carefully derived from the program’s behaviorally-stated learning objectives. These objectives were developed following exhaustive consultation with the appropriate levels of management and staff. Using this research, an instrument to assess the appropriate behavior was issued to the line managers of all of the attendees of the program before the event and six months after the event. Comparisons of pre and post measures provided evidence of behavior change due to the training program. The program participants completed a proprietary transfer system inventory (STAR), which showed how they perceived the factors leading to transfer of training. After the data from both instruments were collected, a regression analysis was conducted to find those few factors that were critical in affecting behavior change. Future workshops are planned to gain more understanding of how these factors operate in ICON. In addition, there are plans to review how sustained and sustainable are the change in behaviors achieved.
Significant learning regarding transfer resulted from this case study. The role of outcomes evaluation measures, such as those levels in the Kirkpatrick and Phillips models, was clarified. These measures are a useful guide to the impact of the program in retrospect. They indicate what the organization is likely to have gained from the execution of the program; however, they are limited in their inability to indicate problems in transfer where the impact of the program was not as expected. This is not a trenchant criticism of these models; they do what they were designed to do.