Search the World's Largest Database of Information Science & Technology Terms & Definitions
InfInfoScipedia LogoScipedia
A Free Service of IGI Global Publishing House
Below please find a list of definitions for the term that
you selected from multiple scholarly research resources.

What is Web Crawler

Handbook of Research on Innovations in Database Technologies and Applications: Current and Future Trends
An automated program used by search engines to collect web pages to be indexed.
Published in Chapter:
Deep Web: Databases on the Web
Denis Shestakov (Turku Centre of Computer Science, Finland)
DOI: 10.4018/978-1-60566-242-8.ch062
Finding information on the Web using a web search engine is one of the primary activities of today’s web users. For a majority of users results returned by conventional search engines are an essentially complete set of links to all pages on the Web relevant to their queries. However, currentday searchers do not crawl and index a significant portion of the Web and, hence, web users relying on search engines only are unable to discover and access a large amount of information from the nonindexable part of the Web. Specifically, dynamic pages generated based on parameters provided by a user via web search forms are not indexed by search engines and cannot be found in searchers’ results. Such search interfaces provide web users with an online access to myriads of databases on the Web. In order to obtain some information from a web database of interest, a user issues his/her query by specifying query terms in a search form and receives the query results, a set of dynamic pages which embed required information from a database. At the same time, issuing a query via an arbitrary search interface is an extremely complex task for any kind of automatic agents including web crawlers, which, at least up to the present day, do not even attempt to pass through web forms on a large scale.
Full Text Chapter Download: US $37.50 Add to Cart
More Results
Deception Detection on the Internet
A program used to search through pages on the World Wide Web for documents containing a specific word, phrase, or topic
Full Text Chapter Download: US $37.50 Add to Cart
Advanced Emergency Response Management in Smart Environments
A software program able to automatically analyse the content of web pages in order to recognize useful information. In the proposed Hybrid Management system, the web crawler is used to detect and extract from social platforms information regarding disaster events. Then, a neural tree-based network is used to automatically classify the information retrieved as belongs to an emergency event or not.
Full Text Chapter Download: US $37.50 Add to Cart
A Survey of Link Analysis Ranking
Also known as “spider,” “robot,” “Webbot,” or “bot.” It is a program that fetches Web pages to a server in order to be indexed probably by another program. It usally “crawls” the Web by following links, but it is also an option to find the URLs that should be fetched in a database.
Full Text Chapter Download: US $37.50 Add to Cart
Twitter Data Mining for Situational Awareness
Software program able to automatically analyze the content of web pages in order to recognize useful information.
Full Text Chapter Download: US $37.50 Add to Cart
Cognitive Systems for Dark Web Cyber Delinquent Association Malignant Data Crawling: A Review
A Web crawler is a Web bot who routinely navigates the World Wide Web and is invariably run by search engines for the purpose of Web indexing. It is also known as a spider, spiderbot, or just a crawler.
Full Text Chapter Download: US $37.50 Add to Cart
eContent Pro Discount Banner
InfoSci OnDemandECP Editorial ServicesAGOSR