Science is on the verge of practical agent based modeling decision support systems capable of machine learning for healthcare policy decision support. The details of integrating an agent based model of a hospital emergency department with a genetic programming machine learning system are presented in this paper. A novel GP heuristic or extension is introduced to better represent the Markov Decision Process that underlies agent decision making in an unknown environment. The capabilities of the resulting prototype for automated hypothesis generation within the context of healthcare policy decision support are demonstrated by automatically generating patient flow and infection spread prevention policies. Finally, some observations are made regarding moving forward from the prototype stage.
TopIntroduction
The time is approaching when agent based models (ABM), machine learning (ML), and combined ABM-ML systems will be increasingly used to undertake or aid in addressing certain complex optimization problems formerly left entirely to expert human decision makers. These decision makers often operate in an ad-hoc mode of incremental improvement to existing policy, in other words an evolutionary optimization process without a formalized closed loop of performance feedback control. Specifically, an ABM-ML decision support system capable of assisting with automated hyptothesis generation is demonstrated. An agent based model is a type of multi-agent system (MAS) in which individual agents (entities capable of making decisions and interacting with their environment and other agents) of varying types along with their environment comprise the modeled system (Uhrmacher & Weyns, 2009). Figure 1 shows the general relationship between ABM and other several other modeling methods.
Figure 1. Comparison of modeling and simulation methodologies
The definition of machine learning for the purposes of this work will be “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E” (Mitchell, 1997, p. 2). The author’s previous work (Laskowski, Demianyk, Naigeboren, Podaima, Friesen, & McLeod, 2010; Laskowski, McLeod, Friesen, Podaima, & Alfa, 2009; Mukhi & Laskowski, 2009) has described a novel ABM used to simulate hospital emergency department processes, centered around the interactions of various agents namely, patients, nurses, and doctors.
This work aims to prototype and demonstrate the utility of a decision support tool based on the aforementioned ABM-ML technology, thereby a step towards “closing the loop” for complex decision making domains. The application presented in this work is within healthcare policy decision support, a context defined by social interactions which are by their stochastic nature difficult to bind. Specifically, a genetic programming (GP) (Koza, Keane, Streeter, Mydlowec, Yu, & Lanza, 2005; Bhanzhaf, Nordin, Keller, & Francone, 1998) ML module was combined with the emergency department (ED) ABM in order to automatically generate policies to optimize patient flow, and to optimize infection mitigation and control policies for contact-based infections such as influenza, and that the quality of these policies improves over time. The policies are evaluated directly in the environment of the simulated ED, collecting several performance metrics for each policy.
TopBackground
Many MAS implementations have been developed with ML capabilities (Uhrmacher & Weyns, 2009), but not in the context of healthcare policy decision support. The novelty of this work is that it automatically optimizes policy or behaviors of agents within a simulated healthcare context. Overall, relatively little work has been done applying machine learning within decision support tools for emergency department management. Two notable examples of basic ML applications within such tools include optimization of staffing schedules using discrete event simulation (DES) and a genetic algorithm (Yeh & Lin, 2007); and training of a neural-network based metamodel for rapid interpolation of ED DES results (Kilmer, Smith, & Shuman, 1997). This work is novel as it is the first instance where a patient treatment or flow policy is automatically generated.