Big Data Challenges
In 2009, the total global amount of stored data was estimated to have reached 800 Exabyte (EB) (Association, 2010) and was increased by approximately 13 EB throughout the following year (Agrawal et al., 2012). It was recently estimated that the amount new data generated in 2013 alone has reached 900 EB, implying that the vast majority of data stored today have been generated in just the past two years (IBM). Of this, the global amount of healthcare data was estimated to have exceeded 150 EB in 2011 (IBM). There is a simple explanation for this strong increase: Data nowadays are generated anywhere and anytime in a mainly automated manner and storing them is relatively cheap (e.g. commercial data storage is offered for less than USD 0.01 per Gigabyte and month) (AWS). The notion of ‘Big Data’ to describe this phenomenon that large amounts of data are generated within a specific domain or of a specific class has already been described in the mid-nineties of the last century when the term itself was first coined by John Mashey of Silicon Graphics (sgi) and since then been widely adopted (Lohr, 2013; Mashey, 1998).
Most types of Big Data have many characteristics in common, e.g. a typical life cycle. They are generated, copied and moved, processed and analyzed, versioned, archived and sometimes deleted. Each step brings up systematic issues, all of them involving IT. Handling of Big Data starts with the process of generation. To generate data and to keep them usable, it is important to document their existence. By whom, how, when and under what circumstances were they created? The lack of such attributes will reduce or even disable the usability of data. Already, this is not trivial; as such annotation should be stored in a searchable manner. A popular tool to handle Big Data, especially for data annotation, tracing and versioning is ‘iRods’ (www.irods.org), which is employed, for example by the CERN (for data from high energy physics experiments) and the Wellcome Trust Sanger Institute (WTSI) (for DNA sequencing data).
Assuming the data have been adequately annotated, they are oftentimes placed within the Internet for immediate global availability, creating the challenge to interested users of acquiring a local copy. Classical transfer methods based on FTP or HTTP were not designed to transfer large files (i.e. in the range of gigabytes or more). Tools like AsperaTM (using a proprietary protocol named FASP) or torrent-seeding based methods (cghub.ucsc.edu) remedy this issue to some extent but are not yet widely spread.
Finally, due to their size, Big Data are quite often stored on distributed architectures, bringing up another issue. Software meant to process Big Data must take into account partial failures of the underlying hardware (e.g. a single disk error) and communication latencies. Many popular software products are in the process of redesign to adapt to this change. To speed up the analysis of Big Data, they are processed in a parallel manner in such distributed environments. This can be done with tools like HadoopTM (Apache, 2012), utilizing a proprietary file system to distribute data across the network and a so-called ‘master-slave approach’ to assign sub-tasks to interlinked compute nodes. HadoopTM is a framework developed by Apache and used amongst others by FacebookTM and YahooTM.