A New Approach for Building a Scalable and Adaptive Vertical Search Engine

Abstract

Search engines are the most important search tools for finding useful and recent information on the Web today. They rely on crawlers that continually crawl the Web for new pages. Meanwhile, focused crawlers have become an attractive area for research in recent years. They suggest a better solution for general-purpose search engine limitations and lead to a new generation of search engines called vertical-search engines. Searching the Web vertically is to divide the Web into smaller regions; each region is related to a specific domain. In addition, one crawler is allowed to search in each domain. The innovation of this article is adding intelligence and adaptation ability to focused crawlers. Such added features will certainly guide the crawler perfectly to retrieve more relevant pages while crawling the Web. The proposed crawler has the ability to estimate the rank of the page before visiting it and adapts itself to any changes in its domain using.