Benchmarking Grid Applications for Performance and Scalability Predictions

Introduction

Grid infrastructures provide an opportunity to the scientific and business communities to exploit the powers of heterogeneous resources in multiple administrative domains under a single umbrella (Foster & Kesselman, The Grid: Blueprint for a Future Computing Infrastructure, 2004). Proper characterization of Grid resources is of key importance in effective mapping and scheduling of the jobs in order to minimize execution time of complex workflows and utilize maximum power of these resources.

Benchmarking has been used for many years to characterize a large variety of resources ranging from CPU architectures to file systems, databases, parallel systems, internet infrastructures, or middleware (Dikaiakos, 2007). There have always been issues regarding optimized mapping of jobs to the Grid resources on the basis of available benchmarks (Tirado-Ramos, Tsouloupas, Dikaiakos, & Sloot, 2005). Existing Grid benchmarks (or their combinations) do not suffice to measure/predict application performance and scalability, and give a quantitative comparison of different Grid sites for individual applications while taking into effect variations in the problem size. In addition, there are no integration mechanisms and common units available for existing benchmarks to make meaningful inferences about the performance and scalability of individual Grid applications on different Grid sites.

Application benchmarking on the Grid can provide a basis for users and Grid middleware services (like meta-schedulers (Berman, et al., 2005) and resource brokers (Raman, Livny, & Solomon, 1999)) for optimized mapping of jobs to the Grid resources by serving as an evaluation of fitness to compare different computing resources in the Grid. The performance results obtained from real application benchmarking are much more useful for scheduling these applications on a highly distributed Grid infrastructure than the regular resource information provided by the standard Grid information services (Tirado-Ramos, Tsouloupas, Dikaiakos, & Sloot, 2005) (Czajkowski, Fitzgerald, Foster, & Kesselman, 2001). Application benchmarks are also helpful in predicting the performance and scalability of Grid applications, studying the effects of variations in application performance for different problem sizes, and gaining insights into the properties of computing nodes architectures.

However, the complexity, heterogeneity, and the dynamic nature of Grids raise serious questions about the overall realization and applicability of application benchmarking. Moreover, diversified types of applications, multi-constrained executions, and heavy computational costs make the problem even harder. Above all, mechanizing the whole process of controlling and managing benchmarking experiments and making benchmarks available to users and Grid services in an easy and flexible fashion makes the problem more challenging.

To overcome this situation, we present a three layered Grid application benchmarking system that produces benchmarks for Grid applications taking into effect the variations in problem size and machine size of the Grid sites. Our system provides the necessary support for conducting controlled and reproducible experiments, for computing performance benchmarks accurately, and for comparing and interpreting benchmarking results in the context of application performance and scalability predictions. It takes the specifications of executables, set of problem sizes, pre-execution requirements and the set of available Grid sites in an input in XML format. These XML specifications, along with the available resources are parsed to generate jobs to be submitted to different Grid sites. At first, the system completes pre-experiment requirements like the topological order of activities in a workflow, and then runs the experiments according to the experimental strategy. The benchmarks are computed from experimental results and archived in a repository for later use. Performance and scalability prediction and analysis from the benchmarks are available through a graphical user interface and Web Service Resource Framework (WSRF) (Banks, 2006) service interfaces. We do not require complex integration/analysis of measurements, or new metrics for interpretation of benchmarking results.