Web Mining by Automatically Organizing Web Pages into Categories

Web Mining by Automatically Organizing Web Pages into Categories

Ben Choi (Louisiana Tech University, USA)
DOI: 10.4018/978-1-60566-144-5.ch012
OnDemand PDF Download:


Web mining aims for searching, organizing, and extracting information on the Web and search engines focus on searching. The next stage of Web mining is the organization of Web contents, which will then facilitate the extraction of useful information from the Web. This chapter will focus on organizing Web contents. Since a majority of Web contents are stored in the form of Web pages, this chapter will focus on techniques for automatically organizing Web pages into categories. Various artificial intelligence techniques have been used; however the most successful ones are classification and clustering. This chapter will focus on clustering. Clustering is well suited for Web mining by automatically organizing Web pages into categories each of which contain Web pages having similar contents. However, one problem in clustering is the lack of general methods to automatically determine the number of categories or clusters. For the Web domain, until now there is no such a method suitable for Web page clustering. To address this problem, this chapter describes a method to discover a constant factor that characterizes the Web domain and proposes a new method for automatically determining the number of clusters in Web page datasets. This chapter also proposes a new bi-directional hierarchical clustering algorithm, which arranges individual Web pages into clusters and then arranges the clusters into larger clusters and so on until the average inter-cluster similarity approaches the constant factor. Having the constant factor together with the algorithm, this chapter provides a new clustering system suitable for mining the Web.
Chapter Preview


Web mining aims for finding useful information on the Web (Scime & Sugumaran, 2007; Linoff & Berry, 2001; Mena, 1999). The first stage of Web mining is searching. search engines, such as Google, focus on searching (Berry & Browne, 1999). Search engines first try to find as many Web pages as possible on the Internet. This is done by Web crawlers, which go from Web pages to Web pages to retrieve as many addresses (URLs) of Web pages as possible. Since current search engines use keyword search, keywords on each Web page found by the Web crawler are stored on databases for fast retrieval (Baberwal & Choi, 2004).

The next stage of Web mining is the organization of Web contents, which is the objective of this chapter. Since majority of Web contents are stored in the form of Web pages, current search engines and most current researches focus on organizing Web pages (Choi, 2001). Search engines, such as Google, focus of ordering Web pages based on the relevance of the Web pages in relating to the search keywords. Some search engines, such as Yahoo, also try to organize Web pages into categories. Yahoo tries to classify Web pages manually by having people read the contents of the Web pages and assign them to categories. Since the number of Web pages on the Internet has grown to the order of several billions, the manual method of classifying Web pages has been proved to be impractical. Thus, most current researches in Web mining focus on automatically organizing Web pages into categories (Choi & Yao, 2005; Yao & Choi 2007).

Various Artificial Intelligence techniques have been used to facilitate the process of automatically organizing Web pages into categories. Two of the most successful techniques are automatic classification and clustering. Web page classification assigns Web pages to pre-defined categories (Choi & Yao, 2005). Since defining a category is not an easy task, machining learning methods have been used to automatically create the definition from a set of sample Web pages (Choi & Peng, 2004). Web page clustering does not require pre-defined categories. It is a self-organization method based solely on measuring whether a Web page is similar to others. It groups Web pages having similar contents into clusters. This chapter will focus on automatic clustering of Web pages.

The organization of Web contents will then facilitate the final stage of Web mining, which is the extraction of useful information from the Web. Nowadays the extraction of useful information from the Web is usually done by search engine users, who have to scan Web pages after Web pages in hope of finding the useful information and often give up without getting the needed information. The results of organizing Web pages into categories or clusters will allow the users to focus on the groups of Web pages that are relevant to their needs.

The future of Web mining is moving toward Semantic Web, which aims for automatically extracting useful information from the Web (Antoniou & van Harmelen, 2004). For a computer to automatically extract useful information from the Web, the computer first needs to understand the contents of Web pages. This is done with the help of natural language understanding and with the help of assigning meaningful tags to strings of characters. For instance, a string of digits may be assigned as phone number or a string of digits and letters may be assigned as address. Understanding of Web contents will also help organizing Web pages into categories and on the other hand the organization of Web contents can facilitate the understanding (Choi & Guo, 2003; Peng & Choi, 2005).

Complete Chapter List

Search this Book:
Editorial Review Board
Table of Contents
Vijayan Sugumaran
Chapter 1
Hong Lin
In this chapter a program construction method based on ?-Calculus is proposed. The problem to be solved is specified by first-order predicate logic... Sample PDF
Designing Multi-Agent Systems from Logic Specifications: A Case Study
Chapter 2
Rahul Singh
Organizations use knowledge-driven systems to deliver problem-specific knowledge over Internet-based distributed platforms to decision-makers.... Sample PDF
Multi-Agent Architecture for Knowledge-Driven Decision Support
Chapter 3
Farid Meziane
Trust is widely recognized as an essential factor for the continual development of business-to-customer (B2C) electronic commerce (EC). Many trust... Sample PDF
A Decision Support System for Trust Formalization
Chapter 4
Mehdi Yousfi-Monod
The work described in this chapter tackles learning and communication between cognitive artificial agents and trying to meet the following issue: Is... Sample PDF
Using Misunderstanding and Discussion in Dialog as a Knowledge Acquisition or Enhancement Procecss
Chapter 5
Sungchul Hong
In this chapter, we present a two-tier supply chain composed of multiple buyers and multiple suppliers. We have studied the mechanism to match... Sample PDF
Improving E-Trade Auction Volume by Consortium
Chapter 6
Manoj A. Thomas, Victoria Y. Yoon, Richard Redmond
Different FIPA-compliant agent development platforms are available for developing multiagent systems. FIPA compliance ensures interoperability among... Sample PDF
Extending Loosely Coupled Federated Information Systems Using Agent Technology
Chapter 7
H. Hamidi
The reliable execution of mobile agents is a very important design issue in building mobile agent systems and many fault-tolerant schemes have been... Sample PDF
Modeling Fault Tolerant and Secure Mobile Agent Execution in Distributed Systems
Chapter 8
Xiannong Meng, Song Xing
This chapter reports the results of a project attempting to assess the performance of a few major search engines from various perspectives. The... Sample PDF
Search Engine Performance Comparisons
Chapter 9
Antonio Picariello
Information retrieval can take great advantages and improvements considering users’ feedbacks. Therefore, the user dimension is a relevant component... Sample PDF
A User-Centered Approach for Information Retrieval
Chapter 10
Aboul Ella Hassanien, Jafar M. Ali
This chapter presents an efficient algorithm to classify and retrieve images from large databases in the context of rough set theory. Color and... Sample PDF
Classification and Retrieval of Images from Databases Using Rough Set Theory
Chapter 11
Lars Werner
Text documents stored in information systems usually consist of more information than the pure concatenation of words, i.e., they also contain... Sample PDF
Supporting Text Retrieval by Typographical Term Weighting
Chapter 12
Ben Choi
Web mining aims for searching, organizing, and extracting information on the Web and search engines focus on searching. The next stage of Web mining... Sample PDF
Web Mining by Automatically Organizing Web Pages into Categories
Chapter 13
John Goh
Mobile user data mining is about extracting knowledge from raw data collected from mobile users. There have been a few approaches developed, such as... Sample PDF
Mining Matrix Pattern from Mobile Users
Chapter 14
Salvatore T. March, Gove N. Allen
Active information systems participate in the operation and management of business organizations. They create conceptual objects that represent... Sample PDF
Conceptual Modeling of Events for Active Information Systems
Chapter 15
John M. Artz
Earlier work in the philosophical foundations of information modeling identified four key concepts in which philosophical groundwork must be further... Sample PDF
Information Modeling and the Problem of Universals
Chapter 16
Christian Hillbrand
The motivation for this chapter is the observation that many companies build their strategy upon poorly validated hypotheses about cause and effect... Sample PDF
Empirical Inference of Numerical Information into Causal Strategy Models by Means of Artificial Intelligence
Chapter 17
Yongjian Fu
In this chapter, we propose to use N-gram models for improving Web navigation for mobile users. Ngram models are built from Web server logs to learn... Sample PDF
Improving Mobile Web Navigation Using N-Grams Prediction Models
Chapter 18
Réal Carbonneau, Rustam Vahidov, Kevin Laframboise
Managing supply chains in today’s complex, dynamic, and uncertain environment is one of the key challenges affecting the success of the businesses.... Sample PDF
Forecasting Supply Chain Demand Using Machine Learning Algorithms
Chapter 19
Teemu Tynjala
The present study implements a generic methodology for describing and analyzing demand supply networks (i.e. networks from a company’s suppliers... Sample PDF
Supporting Demand Supply Network Optimization with Petri Nets
About the Contributors