Key Distributed Components for a Large-Scale Object Storage

Key Distributed Components for a Large-Scale Object Storage

Miriam Allalouf (Jerusalem College of Engineering, Israel), Ghislain Chevalier (Orange Labs, France), Danny Harnik (IBM Haifa Research Labs, Israel) and Sivan Tal (Infinidat Ltd., Israel)
Copyright: © 2013 |Pages: 18
DOI: 10.4018/978-1-4666-3934-8.ch002


This chapter discusses distributed mechanisms that serve as building blocks in the construction of the VISION Cloud object service. Two are fundamental building blocks in the creation of a large-scale clustered object storage. These are distributed file systems and distributed data management systems. In addition, the authors study two complimentary topics that aim to improve the qualities of the underlying infrastructure. These are resource allocation mechanisms and improvements to data mobility via data reduction.
Chapter Preview

Distributed File Systems


File systems evolved over time. Starting with local file systems over time additional file systems appeared focusing on specialized requirements such as data sharing, remote file access, distributed file access, parallel files access, HPC, archiving, etc. A Distributed file System (DFS) is a network file system whose clients, servers, and storage devices are dispersed among the machines of a distributed system or intranet. Using a networking protocol between nodes, a DFS allows a single file system to span across all nodes in the DFS cluster, effectively creating a unified Global Namespace for all files:

  • Unifying files on different computers into a single namespace.

  • Files are distributed across multiple servers appearing to users as being stored at a unique place on the network.

  • Users no longer need to know and specify the actual physical location of files in order to access them.

Distributed file systems may include facilities for transparent replication and fault tolerance. When a limited number of nodes in a file system go offline, the system continues to work without any data loss or unavailability. Distributed File Systems (DFS) maintain control of file and data layout across the nodes and employ metadata and locking mechanisms that are fully distributed and cohesively maintained across the cluster, enabling the creation of a very large global pool of storage. A DFS can seamlessly scale to petabytes of storage. A fully distributed file system can handle metadata operations, file locking, and cache management tasks by distributing operations across all the nodes in the cluster.

Complete Chapter List

Search this Book: