Abstract
The authors discuss the evaluation of highly interactive and novel context-aware system with a methodology based on a TREC-like benchmark. We take as a case study an application for Web content perusal by means of context-aware mobile devices, named Context-Aware Browser. In this application, starting from the representation of the user’s current context, queries are automatically constructed and used to retrieve the most relevant Web contents. Since several alternatives for query construction exist, it is important to compare their effectiveness, and to this aim we developed a TREC-like benchmark. We present our approach to early stage evaluation, describing our aims and the techniques we apply. The authors underline how, for the evaluation of context-aware retrieval systems, the benchmark methodology adopted can be an extensible and reliable tool.
TopContext-Aware Retrieval
With the spread of the concepts related to context-aware computing, Information Retrieval has gained new and increasing importance. The newborn field of CAR, instead of concentrating only on topicality, incorporates contextual information into the retrieval process, aiming at discovering “the query behind the context”: to retrieve what the users need, even if they did not issue any query (Mizzaro, 2008). CAR systems are concerned with the acquisition and understanding of context, and with a behavior based on the recognized context. Thus the CAR model includes, among the elements of the classical IR model, the user's context. This context is both used in the query formulation process and associated with the documents candidate for retrieval.
Typical CAR applications present the following characteristics (Jones, 2004): a mobile user, i.e., a user whose context is changing; interactive or automatic actions, if there is no need to consult the user; time dependency, since the context may change; appropriateness and safety to disturb the user. Although CAR applications can be both interactive and proactive in their communication with the user, we concentrate on the proactive aspects, since they are more relevant to our proposal. Besides, we concentrate on the association between CAR and mobile application, as they can be considered as the prime field for CAR (Jones, 2004).
Key Terms in this Chapter
Context Aware Browser: general-purpose solution to Web content fruition by means of context-aware mobile devices.
Proactive: controlling a situation by causing something to happen rather than waiting to respond to it after it happens
Context-Awareness: refers to the idea that computers can both sense, and react based on their environment, and based on the situation they are used in.
Mobile Devices: a mobile device (also known as cellphone device, handheld device, handheld computer) is a pocket-sized computing device.
Information Retrieval: is the science of searching for documents, for information within documents.
Evaluation: action of determining how a particular system behaves using criteria against a set of standards
Benchmark: in IR a benchmark is composed by a collection of topics (expressions of user's information needs), a collection of resources to retrieve, and a set of relevance judgments about those resources for each topic. It is a kind of evaluation that aims at measuring performances of systems in a controlled environment.