Article Preview
Top1. Introduction
When confronted with a high volume of data records generated by software systems, sensors, social networks, mobiles, etc., we need systems to manage and utilize this huge amount of data more than ever before. Current database management systems (DBMSs) cannot manage this huge amount of information, so a change is needed in this area. For several years, the managing of huge amounts of data has been known as big data management, where data volume is one dimension of big data. Other dimensions include data item veracity, velocity, and variety, which are out of the scope of this paper.
One of the areas needing big data management is data warehousing. When working with a high volume of information generated by online transaction process (OLTP) systems, creating a data warehouse for this huge amount of information is critical. Information retrieval is one of the most important factors in data warehousing. This huge amount of data cannot be stored only on one server, and data must be distributed over several nodes. There are two architectures for distributed solutions: shared memory and storage and shared nothing. In the shared memory and storage architecture, for example, Oracle real application cluster (RAC) servers have a shared memory in storage area network(SAN) storage. They have complex configurations and high maintenance costs. A node count limitation is another big problem. Another group of distributed solutions is that of shared nothing solutions. This group of solutions usually uses a distributed platform to store and retrieve information. One of the most popular groups of shared nothing solutions is not only structured query language (NOSQL), by which structured and non-structured data can be supported. Usually, users’ queries are converted to MapReduce tasks by the NOSQL interface. MapReduce (Dean et al., 2008) is a programming method that can solve big data problems using distributed and scalable solutions. The data entry speed in NOSQL data warehouses is very high because they do not have to observe DBMS constraints. NOSQL data warehouses usually do not have DBMS facilities, such as an index, different data types, etc.
However, the main problem with distributed data warehouses is not DBMS facilities. The main problem is data locality, as it does not exist for data processing on the node, which is needed for processing. Many attempts have been made to conquer this problem, but to the best of our knowledge, there is no method to solve this problem completely.
In this paper, we introduce the Hengam method, which offers a scalable and distributable data warehouse for big data; this method is based on MapReduce. In the proposed method, the data locality problem is solved completely, and traditional DBMSs can be used on distributed nodes. The proposed method was evaluated with the data items of an ICT company.