Commercial and Distributed Storage Systems

Introduction

Distributed storage systems are becoming the method of data storage for the new generation of applications - Web applications by companies like Google, Amazon and Yahoo!. There are several reasons that explain the increasing trend towards distributed processing. On one hand, programs should be scalable and should take advantage of multiple systems as well as multi-core CPU architectures. On the other hand, Web servers have to be globally distributed for low latency and failover. Object systems differ from file systems in the data model and access semantics they provide. Distributed object stores support wide distribution of data and access to data with high availability, even in presence of frequent node failures. They support data storage cloud services in which data is read and written by a wide variety of client applications running anywhere and typically using HTTP interfaces to access the storage service and their data. Several main differences in the characteristics of distributed object stores compared to file systems are:

• •

  Flat namespace. Each object is addressed individually and independently of other objects, eliminating the hierarchical directory structure of file systems. With a flat namespace, each operation affects a single object, which helps keep consistency in widely distributed systems. Object metadata support provides some data management capabilities that compensate for the absence of directories. Some object stores provide the notion of containers, or buckets, that act as collections of objects and divide the namespace into multiple ones, also serving as a means of managing data and isolation between different data sets and multiple clients.

• •

  Fixed content. Objects are typically written as a whole, rather than updated at the byte range level. This also helps keeping data consistency in the presence of multiple geographically distributed writers, while keeping the object stored highly available for read and write.

• •

  Relaxed consistency models. Storage clouds consisting of distributed object stores deploy an eventual consistency model, meaning that when no node and network failures occur, eventually all nodes will have the same view of the data.

• •

  Modern distributed object stores are typically built for world-wide spread with nodes distributed over a WAN. Data redundancy is typically supported through replication rather than RAID. Replicating data across distant locations also improves the availability and latency in access to the data. Each object storage cloud service has its own HTTP based API, and each API evolves over time by adding more features support through the API. The flexible APIs are developer-friendly and allow for rapid enhancements of the service. On the other hand, lack of standardization in this area results in lack of interoperability and the data lock-in problem. Providing storage as a cloud service introduces a new ICT delivery model for storage. Businesses and individuals can consume flexible, variable and unlimited amount of storage without acquiring hardware, managing and maintaining it, paying per use for storage consumed “on demand”.

In this chapter, we first present the requirements end-users pose on cloud storage services. Then we provide a high-level overview of the most prominent commercial offerings in cloud storage and compare them against the requirements set out. Lastly, we briefly present how the innovations of VISION Cloud compare to the presented commercial solutions

Requirements

Within the context of the VISION Cloud project a number of requirements for a storage service have been identified. These can be basically broken down to the following categories: