Multimodal Search on Mobile Devices: Exploring Innovative Query Modalities for Mobile Search

Multimodal Search on Mobile Devices: Exploring Innovative Query Modalities for Mobile Search

Xin Fan (University of Sheffield, UK), Mark Sanderson (University of Sheffield, UK) and Xing Xie (Microsoft Research Asia, China)
DOI: 10.4018/978-1-60566-978-6.ch010
OnDemand PDF Download:
No Current Special Offers


The increasingly popularity of powerful mobile devices, such as smart phones and PDAs, enables users to search for information on the move. However, text is still the main input modality in most current mobile search services although some providers are attempting to provide voice-based mobile search solutions. In this chapter, we explore the innovative query modalities to enable mobile devices to support queries such as text, voice, image, location, and their combinations. We propose a solution to support mobile users to perform visual queries. The queries by captured pictures and text information are studied in depth. For example, the user can simply take a photo of an unfamiliar flower or surrounding buildings to find related information from the Web. A set of indexing schemes are designed to achieve accurate results and fast search through large volumes of data. Experimental results show that our prototype system achieved satisfactory performance. Also, we briefly introduce a prospective mobile search solution based on our ongoing research, which supports multimodal queries including location information, captured pictures and text information.
Chapter Preview

State Of Art

Initially, mobile search services were simple counterpart versions of web search ones and the content was mainly limited to news, weather, sports, etc. In recent years, remarkable efforts have been made on both interface and content of mobile search services in the research community. The search providers are also shifting their focus on multimodal interfaces and richer content types. A location-aware system (e.g. GPS), embedded camera and micro-phone can be successfully exploited to collect multimodal information and generate multimodal queries for better search experiences on mobile devices. In this section, we will specifically study the present research work and commercial search services on mobile audio search, mobile visual search and location based mobile search,

Complete Chapter List

Search this Book: