Data Pre-Processing in Content Mining
Web content mining is strongly related to the domain of Text Mining, since in order to process and organize Web pages and their content should be first appropriately processed in order to extract patterns of interest. These selected properties are subsequently used to represent the documents and assist the clustering or classification processes.
Another essential issue during this stage is semantic analysis. Semantic analysis deals mainly with the problems of synonymy (different names for the same concept) and polysemy (different concepts having the same name). Research on the area of “Word Sense Disambiguation” (WSD) has dealt with this problem. Word sense disambiguation is achieved by assigning words to appropriate concepts. The mapping from words to concepts should be done in a reliable way, depending on the relations between words under examination.
Data Pre-Processing in Web Usage Mining
There are some important technical issues that must be taken into consideration during this phase in the context of the Web personalization process. It is necessary for Web log data to be prepared and pre-processed in order to use in the consequent phases of the process. The first issue is data preparation, depending on the application, Web log data may need to be cleaned from entries involving pages that returned an error or graphics file accesses. Furthermore, crawler activity can be filtered out, because such entries do not provide useful information about the site’s usability. Another problem is with caching. Accesses to cached pages are not recorded in the Web log, therefore such information is missed. Caching is heavily dependent on the client-side technologies used and therefore cannot be dealt with ease. In such cases, cached pages can usually be inferred using the referring information from the logs. Moreover, a useful aspect is to perform page view identification, determining which page file accesses contribute to a single page view.
Most important of all is the user identification issue. There are several ways to identify individual visitors. The most obvious solution is to assume that each IP address (or each IP address/client agent pair) identifies a single visitor. Nonetheless, this is not very accurate because, for example, a visitor may access the Web from different computers, or many users may use the same IP address. Again, a user that uses more than one browser, even on the same machine, will appear as multiple users. A further assumption can then be made, that consecutive accesses from the same host during a certain time interval come from the same user. So, identification of user’s browsing behavior is also one important problem during data pre-processing. More accurate approaches for a priori identification of unique visitors are the use of cookies or similar mechanisms or the requirement for user registration. However, a potential problem in using such methods might be the reluctance of users to share personal information. Assuming a user is identified, the next step is to perform session identification. Before any mining is done on web usage data, sequences of page references must be grouped into logical units representing web transactions or user sessions. A user session is all of the page references made by a user during a single visit to a site. Identifying user session is similar to the problem of identifying individual users as discussed above.
Thus some complex processes are used in order to extract knowledge about users of web site. Pre-processing is therefore is crucial issue for learning.