Abstract
Data is an integral part of most business-critical applications. As business data increases in volume and in variety due to technological, business, and other factors, managing this diverse volume of data becomes more difficult. A new paradigm, data virtualization, is used for data management. Although a lot of research has been conducted on developing techniques to accurately store huge amounts of data and to process this data with optimal resource utilization, research remains on how to handle divergent data from multiple data sources. In this chapter, the authors first look at the emerging problem of “big data” with a brief introduction to the emergence of data virtualization and at an existing system that implements data virtualization. Because data virtualization requires techniques to integrate data, the authors look at the problems of divergent data in terms of value, syntax, semantic, and structural differences. Some proposed methods to help resolve these differences are examined in order to enable the mapping of this divergent data into a homogeneous global schema that can more easily be used for big data analysis. Finally, some tools and industrial examples are given in order to demonstrate different approaches of heterogeneous data integration.
TopBackground: Business Analytics With Their Challenges
Over 90% of Fortune 500 companies have a Big data initiative this year. An IBM study has discovered that companies which use Big data analytics perform better than those who do not. (Leavitt, 2013) However, until legislative changes occur, certain industries, such as Finance and Healthcare, are currently required to keep all of their data in-house (Leavitt, 2013).
Analytics provides up-to-the-minute business insights, which have been derived from business data, which helps manage business risks and reduce compliance penalties. (Composite Customer Value Framework, 2012) However, the growing volume and complexity of business data increases business risks and reduces business agility in responding to new threats and opportunities. (Data Virtualization Platform Maturity Model, 2012) Notably, there is a rise in semi-structured data from Web services and non-relational data stores which must be integrated and analyzed for business insights. (Turbo Charge Analytics with Data Virtualization, 2013) Data access and integration pose the biggest bottleneck for analytics. (Data Virtualization Applied, 2012) An example, when a business is analyzing a typical marketing campaign, they must integrate and analyze diverse data from multiple sources: Website click statistics for their marketing Web site, email responses for leads, revenue feeds from Web services, et al.
The data are diverse:
Key Terms in this Chapter
Document Type Definition (DTD): A document that defines the structure of an XML document with a listing of its permissible elements and attributes. A group of elements may be nested within another in a DTD. For example, the elements of FirstName and LastName may be enclosed within another element: PersonRecord. The elements may have attributes that indicate whether the element is required to be present in the document or if it is implicit.
Ontology: An abstract yet systematic description of the information contained in an object within a self-declared domain. Ontologies are often used, in databases, to describe their data with their relations and how they correspond to similar data entities in other databases.
Attribute: An individual characteristic holding a value of data. For example, a data attribute may be a single column, FirstName, holding the value “Tom,” of a particular row or record.
Metadata: Data that are used to describe other data. For example, FirstName is metadata that describes a given set of data, the first names of individuals, within a Person table.
Syntax: The rules governing the use of elements to combine to form a meaningful statement. For example, in SQL (Structured Query Language), a Select statement, which is used to retrieve data, must begin with the SELECT keyword followed by a list of attribute names of data that it wishes to be retrieved.
Extensible Markup Language (XML): A text-based data transfer protocol that consists of metadata tags that enclose data values. These tags can enclose other tags to form a nested structure. An example, <Person><Name>John Doe</Name></Person> indicates a Person record with a field of Name whose value is John Doe.
Schema: The structure of a database that formally defines the tables, columns in each tables, and the relationships between columns of different tables.
XML Schema: Similar to a DTD, an XML Schema defines a list of permissible elements and the structure of an XML document. However, an XML Schema also defines the data types, default values, and non-null/null validity of elements. In addition, an XML Schema can indicate which elements are child elements, and for these child elements, it defines their order and number. It also describes clearer relationships and allows inheritance.