Design of a Novel Search Engine for Prospective Question Answering

1. Introduction

Search engines, in Arasu et al. (2001) and Brin et al. (1988) provide users a way to find information in an organized fashion. A search engine uses special software called a crawler, given in Burner et al. (1997) and Yates et al. (2004) to go through the entire Web and download Web pages using a list of seed URLs. The downloaded Web pages are then stored in a repository and are scanned to extract the URLs existing in them. These URLs are further added to the list of seed URLs for further crawling. After crawler downloads pages, they are then stored in repository and are then passed to another program called indexer, given in Arasu et al. (2001) and Brin et al. (1988) for indexing. Indexer (in nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf) maintains an index that maps keywords to the documents in which they appear. The index is then searched for the answering user’s query.

Yates et al. (1999) discussed that the search engines retrieve the information in the form of documents. In case if the user needs an answer to a question, the user has to go through the entire document to extract the answer to his question, which is a time consuming process. So there is a need to build a system which is capable of taking user’s question as input and returns answer(s) as the result. Hence, the need of Question answering system arises.

Question answering offers a more intuitive approach to information processing. Given a collection of documents and a natural language question posed by user, a question answering system attempts to find the precise answer or at least a portion of the text in which the answer appears. Architecture of a general Question answering system, in Zheng et al. (2002) has been shown in Figure 1. In general, a Question answering system consists of the following modules:

Figure 1.

Architecture of question answering system

• •

  Question-Analysis Module: Identifies the question type, answer type, topic and focus of the question and transforms question into a query.

• •

  Document-Processing Module: Identifies a set of documents that are relevant to user’s query. Also, it filters the paragraphs that are likely to contain answer and rank them.

• •

  Answer-processing Module: Extracts answers from ranked text segments obtained from the above module. The extracted answers are then ranked. The user is thus provided with a list of answers sorted in order of their relevance.