Article Preview
Top1. Introduction
World Wide Web is growing very rapidly from few thousand pages to more than two billion pages at present. The use of web has elevated significant social concerns in the fields of privacy, censorship and access to information (Pinkerton, 2000). Web data consists of enormous number of web pages and links connecting them. Due to its size and dynamic nature it becomes highly difficult to locate the information efficiently across the web. Hence search engines play a vital role in retrieving the relevant and accurate data from the web (Buttcher & Clarke, 2010) and they rely upon the huge collection of web pages obtained by the web crawler, a computer program which browses the World Wide Web in a completely ordered and automated fashion.
Mining for specific data using sequential information retrieval algorithms is time consuming and inefficient. Hence there is a need to develop more parallel algorithms to search and retrieve the user specific data from the ever expanding World Wide Web. The aim of this paper is to utilize a MapReduce framework to build a crawler, indexer and ranking system to retrieve information from the Web and cater to the personalization of the data using Machine learning concepts. The problem with the retrieved results is the order in which they are displayed, as millions of results are obtained it becomes difficult for the user to locate the relevant one. Hence machine learning concept is used to identify the most popular items from the Web and rank them so that it becomes easier for the user to retrieve the popular and important information from the Web (Croft, Metzler, & Strohman, 2010).
PageRank (Brin & Page, 1998) is a link analysis algorithm that assigns a numerical weighting (Qin, Liu, Xu, & Li, 2007) to each element of a hyperlinked set of documents, such as the World Wide Web, with the purpose of “measuring” its relative importance within the set. PageRank hypothesizes that a link from page A to B implies the author of A recommends having a look at page B. A page is said to have a high PageRank value if there are many pages that point to it (Kim & Lee, 2002; Shenoy et al., 2004). The PageRank algorithm and implementation details are explained in detail in the later section. The algorithm represents the structure of the web as a matrix and PageRank as a vector. The PageRank vector is derived by computing matrix-vector multiplications repeatedly. The sum of all PageRank values should be one during computation. In this paper we use MapReduce framework to compute the ranks of the web pages correctly. Experimental evaluation is carried out and its results are briefly discussed.
The objective of this paper is to develop a web portal application with searching capability that runs as a website. As an outcome, the paper also aims at testing the performance of the proposed algorithms on various data sets. Parameters like retrieval time, computation time while ranking of web pages based on popularity and others are tested. The remainder of the paper is organised as follows. Section 2 discusses the related work and Section 3 gives the contributions of the paper. Sections 4 and 5 contain the details of the architectures of crawler, indexer and the ranking system proposed algorithms followed by the pseudocode of all the techniques used. An illustrative example is given in Section 6. Experimental results including timings of the iterative methods on graphs are discussed in Section 7 followed by conclusions.