Modeling Search Systems by Composite Inverted Index

Introduction

In computer science computational work is spoken in terms of effort or elegance and to address this a construct was imposed and it is called Big O notation. In Big O notation what is always sought is constant time, so the questions begs to be asked in the search space; how can this be achieved with searching? The profession has come from the days of doing mass searching over some textual domain by looking at the individual word elements and matching them to the required needs and from these humble beginnings teams have moved to the domain of big data and distributed systems where the search domain is simplified and complicated all at the same time. Take the case of HADOOP and the Hadoop distributed file system (HDFS) where work effort is distributed to agent nodes and the search effort is consequently offset from some primary to a series of child nodes. The profession has regressed back to the beginning of time where 16MB machines searched the hard drive one file at a time, but now it happens in the cloud of course so it must be different. Individuals have taken the problem frame and regressed to something that is known and is comfortable with to put the world back into homeostasis, but in doing so teams have not made that large leap forward in the paradigm. What is shown in this body of work is that for search there are some central tenants that hold and whether the domain of inspection is structured, semi-structured or unstructured data there are parallels and rules to adhere to. By following this prescribed doctrine over the domain it becomes possible to approximate O(1), which is achieved by way of a composite inverted index as is defined in this body of work.

The language construct is what governs the search input into some model that has the burden of determining the truth proposition from those inputs. It is the context of the language construct that also places yet another burden on the determination of worth on the system as for example the search term ‘Big Apple’ will probably refer to the city in the state of New York and would rather not imply a disproportionate fruit. To help alleviate some of the burden of context around these search systems it therefore becomes a necessity to be able to utilize the Natural Language Toolkit (NLTK) that may be viewed here: https://www.nltk.org/. Humanity fundamentally uses language differently to express its thoughts, hence the reason why there exists synonyms, it therefore becomes necessary to incorporate such a component in modeling efforts that compensates for this so that models can be better able to evaluate that truth proposition. The equation developed for the Yahoo search engine proves this point, it’s an artifact that helps understand the landscape that is laid bare before all. It is possible to utilize a language construct along with an indexing effort to approximate truth, this was proved. The question now remains, can this go a step further to create generic search systems that allow for the indexing effort to encompass semi-structured, structured, and non-structured data so that a search system can be created over the entire data landscape? The argument of this chapter is that it can be done, you can create search systems that index data lakes for example where there exists flat files (unstructured data), XML documents (semi-structured data), and database files (structured data) to consequently create search systems that go beyond the myopic view of the businesses data. This chapter defines a design pattern that is referred to as a ‘Composite Inverted Index’ that may be used to create search systems over divergent data store types that when implemented can create solutions to facilitate the search efforts in an efficient and domain specific manner.

The discussion parts by looking at the state of current indexing doctrine that is later expanded on to create a system that can be implemented across industries and businesses. As was the case with the modeling of the Yahoo and Bing search engines, no two are identical; no two business are exactly the same and as such a design pattern must be just that a generic solution that can be implemented across instances as is the case of the interface design pattern for example. The interface design pattern is not programming language specific but is rather indifferent to its implementation. This is the breadth that is needed with solutions, courses of action that surpass the immediate and extend to the generic.