A New Adaptive Indexing for Real-Time Web Search

Introduction

When Brin and Page (Brin and Page, 1998) wrote a seminal “The Anatomy of a Large-Scale Hyper Textual Web Search Engine” they added an appendix about the scalability of Google in which they argued that scalability is limited by their choice for a single and centralized index. While these limitations would decrease over time, following Moore’s law, a truly scalable solution would require a drastic redesign. They wrote the following: “Of course a distributed systems like Gloss or Harvest will often be the most efficient and elegant technical solution for indexing, but it seems difficult to convince the world to use these systems because of the high administration costs of setting up large numbers of installations. Of course, it is quite likely that reducing the administration cost drastically is possible. If that happens, and everyone starts running a distributed indexing system, searching would certainly improve drastically.” Adaptive indexing algorithms exist today are a predictive solution to solve that difficulties. Traditional approaches to build adaptive indexing in distributed networks do not scale well. Traditional approaches, such as search engines or topical directories on the Web have difficulties for locating and indexing rich data (Ambite and Knoblock, 2000). Some academic sectors, e.g. Web search track in Text Retrieval Conference (TREC)¹ help researchers to improve their web search algorithms by using already crawled data. TREC Web track organizers exploited subset B (50 million documents) to generate a new version of research track called “Federated Search” (FedWeb)². The task executes multiple search queries simultaneously in offline resources determined previously, and then, the retrieved results were combined in a coherent list. However, the goal of the track is to evaluate search approaches based on federated searches at a highly scale of data in a realistic setting (Zhao et al., 2015). That also means the FedWeb promotes models on federated searches with realistic Web data. An important aspect of this task was to determine the verticals of data types in a specific category for each query, whereas our proposal predicts the quality of different verticals selected for a particular query (e.g., sports, news, images, etc.). When set of relevant verticals selected for a particular topic, the selected vertical might contain similar or different topics. The second task, which is also challenge, is to determine the resources for each vertical by predicting the quality of the individual resources based on the tested queries. For example, if a user submits a query ‘trombone’, the system should retrieve results regarding adhoc markets rather than informational articles. The system is required to rank all resources in its verticals by the selected topic. Finally, the system merges the results in the optimized list, in which, the ranked list holds titles, URLs and snippets. The difficulty in result merging is not only to calculate the relevance of individual results, but also, to take the diversity of the verticals into account.