Article Preview
TopIntroduction
Managing growing repositories of unstructured or semi structured documents in an organization is becoming increasingly difficult. The size and number of online and offline documents is increasing exponentially. The need for identifying groups of similar documents has also increased for either getting rid of multiple versions of same document or extracting relevant set of documents from huge document repositories. It benefits many applications such as finding near duplicated web pages, replicated web collections, detecting plagiarism. Web search engines are highly benefited as it can be used for focused Crawling. Forming group of documents is not the only challenge, but there is need to identify the relevant group for a newly arrived document. It can be achieved through feature selection mechanism. It means that if the features of newly arrived document can be identified and matched with the feature set for each group of documents from the existing corpus, the new document gets placed in its relevant group depending on the match found (Moon et al., 2013).
Clustering techniques are available and can be readily applied on data in flat file format. Document data is converted to flat file format by extracting the terms from the documents, so documents represent rows and terms are placed in columns (Gulic et al., 2013). The terms are in large number which causes the problem of dimension curse and decreases algorithm efficiency.
To reduce these terms some feature selection technique should be used. TF-IDF (Term Frequency-Inverse Document Frequency) technique is used which extracts only most relevant terms depending on term occurrence frequency and eliminates the most common terms in the corpus (Albitar, et al., 2014).
Problem with TF-IDF is that, it does not consider synonyms of terms (Gulic et al., 2013). Synonyms are important part while presenting a document. Ideal documents do not repeat the same word. In fact multiple synonyms of a single word are mostly used. So term frequency of the term gets reduced and term is not selected by TF-IDF method. Important terms thus get ignored and logically relevant documents fall into different clusters. Sometimes documents on the same topic may get grouped in different clusters.
The proposed approach is similar to other researchers in the way document set is preprocessed to remove noisy and less useful data. Some researchers extend the bag of words (Jashki et al., 2009) by adding the synonyms which affects cluster quality due to increase in dimensions. In the proposed approach synonyms of each term are treated as a group of terms and group frequency is computed as the sum of the degree of similarity to the parent attribute of each term in the group occurring in the document. It changes the major terms/features because individual count increases unlike in TF-IDF. By applying hierarchical agglomerative clustering algorithm, the clusters are obtained at several levels of hierarchy. The feature set of the documents in clusters at the lowest level of hierarchy are iteratively extracted. The newly discovered features are added to extend the feature set at the topmost level.
It results into extended set of features with respect to each group and even wrongly placed document gets the right cluster. After some iteration, algorithm converges and produces stable set of features. Number of iterations and features depend on variance in data set.
The paper is organized as follows. The background presents the relevant work of other researchers on the topic. The next section presents our approach. The experimental set up and results obtained are presented to validate efficacy of the method. Lastly a real world application is presented to highlight applicability of the method. The paper ends with conclusion and future directions.