Document Summarization Using Sentence Features

Preprocessing

Initially the document is segmented into sentences and words for each sentence are extracted. Then the functional words or stop words like “a”, “the”, “of” (frequently occurring insignificant words) are removed from the word list. The words remaining in the sentences are stemmed.

Feature Extraction

Feature is one of the important aspects of any text mining. Therefore the following features for each sentence need to be prepared for input to the optimization model (Table 1).

• 1.

  ft1= Title Feature: It is the similarity between this sentence & the document title. The score of ft1 is calculated as follows:

  
  (1)

• 2.

  ft2= Sentence Length: This feature is employed to penalize sentences that are too short, since these sentences are not expected to belong to the summary. We use longest sentence length of the sentence for normalization. Where i =1…TNS (2)

• 3.

  ft3= Average Sentence Weight: This feature specifies the weight of each sentence by taking term frequency into an account. The score of ft3 is calculated as follows: Where i=1…TNS (3)

• 4.

  ft4= Sentence Position: We assume that the first sentences of a paragraph are the most important. Therefore, paragraph sentences are ranked according to its position in the paragraph and considering the range between 0 to1. For instance, the first sentence in a paragraph has a score value 1; the second has reduced with some value and so on. The score of ft4 is calculated as follows:

  
  Where i=1…TNS (4)

• 5.

  ft5= Inter-Sentence Similarity: It is the similarity between one sentence & other sentences in the document. The score of ft5 is calculated as follows:

  
  (5)

• 6.

  ft6= Proper Noun: It is specifying the sentence inclusion of name entity i.e. the sentence containing proper noun is an important one and it is most probably included in the document summary. The score of ft6 is calculated as follows:

  
  (6)

• 7.

  ft7= Thematic Word: The most frequent words are defined as thematic words. Sentence scores are functions of the thematic words’ frequencies. The score of ft7 is calculated as follows:

  
  (7)

• 8.

  ft8= Numerical Data: Usually the sentence containing numerical data is an important one and it is most probably included in the document summary. The score of ft8 is calculated as follows:

  
  (8)