On the Performance Evaluation of IaaS Cloud Services With System-Level Benchmarks

On the Performance Evaluation of IaaS Cloud Services With System-Level Benchmarks

Sanjay P. Ahuja (University of North Florida, Jacksonville, USA) and Niharika Deval (University of North Florida, Jacksonville, USA)
Copyright: © 2018 |Pages: 17
DOI: 10.4018/IJCAC.2018010104
OnDemand PDF Download:
No Current Special Offers


Infrastructure-as-a-service is a cloud service model that allows customers to outsource computing resources such as servers and storage. This article evaluates four IaaS cloud services - Amazon EC2, Microsoft Azure, Google Compute Engine and Rackspace Cloud in a vendor-neutral approach with regards to system parameter usage including server, file I/O and network utilization. Thus, system-level benchmarking provides objective comparison of cloud providers from performance standpoint. Unixbench, Dbench and Iperf are the System-level benchmarks chosen to test the performance of server, file I/O and network respectively. In order to capture the variation in performance, the tests were performed at different times on weekdays and weekends. With each offering, the benchmarks are tested on different configurations to provide an insight to the cloud users in selection of provider followed by appropriate VM sizing according to the workload requirement. In addition to the performance evaluation, price-per-performance value of all the providers is also examined and compared.
Article Preview

1. Introduction

Cloud computing enables on-demand network access to the collective pool of configurable physical resources such as servers, storage and networks, that can be swiftly provisioned with negligible management effort and minimal cost. Cloud is implemented on two-tier technology: deployment models (Private cloud, Public cloud, and Hybrid cloud environment) and delivery services (Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS)). This study focuses only on public IaaS clouds. IaaS is the lowest tier in the service model stack and delivers the infrastructure equipment including storage, processing, and networking. The IaaS users rent the computing resources from the cloud providers and are responsible for managing applications, data, run time, middleware and operating systems on self-service virtual machines. Small medium businesses (SMBs) and startup companies benefit from public IaaS cloud adoption due to business agility and low infrastructure costs.

Organizations need to comprehensively assess the technical and business requirements of an application. Once the application requirements are thoroughly analyzed, the next step is to choose the appropriate cloud provider against the established criteria. The cloud deployment decision implicitly depends on the provider's capabilities. But cloud industry has saturated with many IaaS cloud vendors and thus selection of appropriate provider from multiple options takes great deal of time. Each cloud provider has diverse service portfolios and pricing structures. In addition to this, the lack of standard approaches and lack of transparency in IaaS industry makes it even more challenging to select appropriate cloud provider.

There are many factors including SLA, security, geographical location that needs to be considered as part of provider selection process. Along with these factors, performance needs to be taken into account as it affects the productivity of an application and annual operational costs. Therefore, the performance capabilities of cloud providers must be evaluated and compared using standard measures. This can be achieved by benchmarking the performance characteristics of cloud providers at system-level so as to establish the baseline performance expectations.

The study chooses four leading public IaaS cloud services - Amazon EC2, Microsoft Azure, Google Compute Engine, and Rackspace Cloud services, to apply to widest possible audience. System-level benchmarks - Unixbench, Dbench and Iperf are used to evaluate the server performance, file I/O performance and network throughput respectively. Data driven approach from the performance and price standpoint is followed to achieve standardized comparison of IaaS services.

The rest of the paper is structured as follows. Existing research work is discussed in the next section. Section-III elaborates on benchmarks used and research methodology. The major findings of the study are discussed in Section IV. Finally, the paper concludes with the recommendations for future work in Section V.

Complete Article List

Search this Journal:
Volume 13: 1 Issue (2023)
Volume 12: 4 Issues (2022): 2 Released, 2 Forthcoming
Volume 11: 4 Issues (2021)
Volume 10: 4 Issues (2020)
Volume 9: 4 Issues (2019)
Volume 8: 4 Issues (2018)
Volume 7: 4 Issues (2017)
Volume 6: 4 Issues (2016)
Volume 5: 4 Issues (2015)
Volume 4: 4 Issues (2014)
Volume 3: 4 Issues (2013)
Volume 2: 4 Issues (2012)
Volume 1: 4 Issues (2011)
View Complete Journal Contents Listing