EUBrazilCC Federated Cloud: A Transatlantic Multi-Cloud Infrastructure

Introduction

It is largely recognized today that many recent scientific advances have only been possible because of the increasing ability of researchers to draw upon information technologies (IT) in a comprehensive and efficient way. The use of these technologies allows them to process massive amounts of data and execute increasingly sophisticated simulation models, generating thereby new knowledge on an unprecedented scale. The intensive use of IT to generate new scientific knowledge is called e-Science (Hey, 2009).

Cloud computing technologies provide greater efficiency, agility and innovativeness in the delivery of IT services by facilitating economies of scale, sharing of irregularly used resources, and adoption of advanced approaches towards resource management (Armbrust et al., 2010). It is therefore not surprising that many e-science initiatives are currently considering the use of cloud computing to support many kinds of scientific activities (Yang et al., 2014).

The EU-Brazil Cloud infrastructure Connecting Federated Resources for Scientific Advancement (EUBrazilCC) is a R&D project that aims to provide a user-centric test bench enabling European and Brazilian research communities to test the deployment and execution of scientific applications on a federated intercontinental e-infrastructure. The resulting e-infrastructure exploits existing resources that include virtualized data centers, supercomputers, and even opportunistically exploited desktops spread out over a transatlantic geographic area. These heterogeneous resources are federated by means of a middleware that provides the necessary features to achieve the challenging established goals.

The objective of this chapter is to describe the architecture, implementation and deployment of the EUBrazilCC federated e-infrastructure, which has been developed according to the requirements of the following three complex applications involving three different scientific areas:

• •

  Leishmaniasis Virtual Laboratory: An application integrating and processing genomic and clinical data from Leishmania and its vectors.

• •

  Cardiovascular Simulation Service: An application integrating a cloud-based simulator of the human arterial vascular system with an HPC-based heart simulator.

• •

  Climate Change and Biodiversity Scientific Gateway: A platform for the integration and processing of big data in order to study the interaction between biodiversity and climate change.

Although the design decisions were driven by the particular needs of these three applications, the resulting solutions are intended to be general enough to be useful also for many other similar types of scientific applications, and the tools used to build the federated e-infrastructure – some of them developed within the project – should be useful also to other distinct e-Science initiatives.

With the end of meeting the requirements of these three applications, each one demanding the integration of expertise, data and computing resources distributed in some cases over the two continents, we decided to adopt a user-centric approach in the development of a federation of private clouds and other computing resources belonging to different partners involved in the project. In order to support the federation of private clouds, a new middleware called fogbow¹ was developed within the EUBrazillCC project.

However, since not all applications can be executed efficiently in the cloud, the EUBrazilCC e-infrastructure also incorporates HPC clusters. These resources are different from those typically obtained from cloud providers, not only with regard to the processing and communication capabilities they provide but, most importantly, in the way that they may be accessed. When interacting with cloud providers, users and applications commonly have access to virtual machines over which they have full control. In contrast, HPC clusters are accessed via specific job submission services that may vary from one HPC provider to another. In the EUBrazilCC federated e-infrastructure, uniform access to HPC resources is enforced by a standard API provided by the CSGrid middleware (de Lima et al., 2006). Both this API and fogbow will be presented in detail below.