A Model for Estimating the Savings from Dimensional vs. Keyword Search

Introduction

People spend a tremendous amount of time searching for information. One estimate puts the average employee’s time at 3-1/2 hours a week for unsuccessful searches (Ultraseek, 2006). For a 1,000 employee company, that works out to $9.7 million a year for just the cost of salary (Ultraseek, 2006). Some estimates put the cost as high as $33 million annually per company when taking into consideration the costs of recreating the information not found (Thompson, 2004). Furthermore, between 60-80% of queries over the intranet (as opposed to the Internet) are for material that the searcher has previously seen (Mukherjee and Mao, 2004).

Keyword search has several well-known problems (for a review, see Blair, 2002), but its advantage over other methods is that once the documents have been saved, there is no additional work that the user has to perform. One alternative to keyword search is dimensional search. Dimensional search eliminates the ambiguity of words (which causes so many of the problems for keyword search) through the use of pre-defined categories (dimensions) to define documents as well as finite sets of possible values for each category. It has been demonstrated that dimensional search reduces the number of irrelevant documents returned in the result set (LaBrie, 2004). However, there is a significant, up-front, time investment that has to be made for dimensional search. In particular, meta-data must be stored about each document, and much of this information must be determined and entered by a human user. So the question becomes, is the increased retrieval accuracy worth the initial cost of categorizing documents?

The content management market was estimated to be over $1 billion in 2003 (Dunwoodie, 2004), and to have grown 9.7% in 2006 (Webster, 2007). Vendors of this software make quite amazing claims about the efficacy of their software, yet for all the money being spent by companies, there has been little academic work done to evaluate these systems. We want to determine the cost, in time, of performing a keyword search versus the cost, in time, of performing a dimensional search, including the initial time-investment. Factors that affect the overall cost of searching include the start-up costs of any content management system, the size of the library (it is much easier to exhaustively search a small library than a large library), the size of the documents in the library (books are more difficult to search than are e-mail messages), and the cost of not finding the document.

While evaluating the best approach to studying this question, we considered a number of research methodologies. A case study approach to this problem, which is largely what IDC, Gartner and other commercial information providers use, would be hampered by a lack of generalizability. Also, attempting to collect data on an employee’s search could be considered invasive by the employee. If employees know that their time and actions are being tracked, they might elect to perform searches outside of such data collection, out of concern that the collected data might be used to evaluate their work rather than the content management software. Moreover, drawing data from a survey of content management product users makes comparison of such data difficult as the nature of searches might vary considerably by company as well as by user. And there is the additional concern that users might not have an accurate sense of the time or the effectiveness of their searches.

An experiment would need to consider all the above factors, plus ensure the proper motivation of the users. For these reasons, we elected to use an analytical modeling approach, which allows us to use different values for variables and examine their impact on search cost. From our model we were able to determine the break-point, in terms of the number of searches, at which dimensional search becomes more cost effective than keyword search. That is, we were able to determine the number of searches an organization must do in order to justify the up-front cost of determining and entering the metadata that is required to support dimensional search.

The rest of this paper is organized as follows. In the next section, we briefly review some of the most important findings in word frequency distributions. After that, we present the basic model for net search cost. We then present a model for estimating the net search cost of keyword searches, followed by a model for estimating the net search cost of dimensional searches. The output of the two models is then compared, followed by a discussion of the implications of the results and possible refinements of the model.