Science gateways, such as the Virtual Imaging Platform (VIP), enable transparent access to distributed computing and storage resources for scientific computations. However, their large scale and the number of middleware systems involved in these gateways lead to many errors and faults. This chapter addresses the autonomic management of workflow executions on science gateways in an online and non-clairvoyant environment, where the platform workload, task costs, and resource characteristics are unknown and not stationary. The chapter describes a general self-management process based on the MAPE-K loop (Monitoring, Analysis, Planning, Execution, and Knowledge) to cope with operational incidents of workflow executions. Then, this process is applied to handle late task executions, task granularities, and unfairness among workflow executions. Experimental results show how the approach achieves a fair quality of service by using control loops that constantly perform online monitoring, analysis, and execution of a set of curative actions.
Top1. Introduction
Distributed computing infrastructures such as campus clusters, Grids, and now Clouds have become daily instruments of scientific research. As collections of independent computers linked by a network presented to the users as a single coherent system (e.g. Open Science Grid, XSEDE, EGI, and Amazon EC2), they enable easy collaboration among researchers, enhanced reliability and availability, and high-performance computing (Romanus et al., 2012). Increasingly, these systems are becoming more complex, heterogeneous, and prone to failures that can affect the productivity of their users.
In the meantime, science gateways, such as the Virtual Imaging Platform (VIP) (Ferreira da Silva et al., 2011; Glatard et al., 2013), are emerging as user-level platforms to facilitate the access to distributed computing and storage resources for scientific computations. Their high-level interface allows scientists to transparently run their analyses on large sets of computing resources. However, their large scale and the number of middleware systems involved in these gateways lead to many errors and faults. Applications running on these infrastructures are also growing in complexity and volume. Moreover, many scientists now formulate their computational problems as scientific workflows (Taylor, 2007). Workflows allow researchers to easily express multi-step computational tasks, for example: retrieve data from an instrument or a database, reformat the data, and run an analysis. Scientists expect such gateways to deliver high quality of service (QoS), where the workload and resources are efficiently and automatically handled, and the system is fault-tolerant. In order to provide fair QoS, scientific workflow executions are often backed by substantial human intervention that requires constantly monitoring of the running experiments and infrastructure to prevent or handle faults and ensure successful application completion.
Automating fault prevention, detection, and handling is challenging in such platforms. Science gateways have no a-priori model of the execution time of their applications because
- 1.
Task costs depend on input data with no explicit model, and
- 2.
Characteristics of the available resources depend on background load (Ferreira da Silva, Juve, et al., 2013).
Modeling application execution time in these conditions requires cumbersome experiments, which cannot be conducted for every new application in the platform. As a consequence, such platforms operate in non-clairvoyant conditions, where little is known about executions before they actually happen. Such platforms also run in online conditions, i.e. users may launch or cancel applications at any time and resources may leave at any time too.
In this chapter, we propose a general self-management process for autonomous detection and handling of operational incidents in scientific workflow executions on grids (Ferreira da Silva, Glatard, & Desprez, 2012, 2013b). Our process is described as a MAPE-K loop (Kephart & Chess, 2003), which consists of monitoring (M), analysis (A), planning (P), execution (E), and knowledge (K). Self-management techniques, generally implemented as MAPE-K loops, provide an interesting framework to cope with online non-clairvoyant problems. They address non-clairvoyance by using a-priori knowledge about the platform (e.g. extracted from traces), detailed monitoring, and analysis of its current behavior. They can also cope with online problems by periodical monitoring updates. Our ultimate goal is to reach a general model of such a scientific gateway that could autonomously detect and handle operational incidents, and control the behavior of non-clairvoyant, online platforms to limit human intervention required for their operation. Performance optimization is a target but the main point is to ensure that correctly-defined executions are completed, that performance is acceptable, and that misbehaving runs (e.g. failures coming from user errors or unrecoverable infrastructure downtimes) are quickly detected and handled before they consume too many resources.
Top2. Background
In this section, we introduce definitions necessary to understand the rest of this chapter, and present the relevant work regarding strategies to address the operational incidents proposed in this chapter: task replication, task grouping, and fairness among workflow executions.