Requirements to a Search Engine for Semantic Multimedia Content

Requirements to a Search Engine for Semantic Multimedia Content

Lydia Weiland, Felix Hanser, Ansgar Scherp
DOI: 10.4018/ijmdem.2014100104
(Individual Articles)
No Current Special Offers


The authors investigate user requirements regarding the interface design for semantic multimedia search and retrieval based on a prototypical implementation of a search engine for multimedia content on the web. Thus, unlike existing image or video search engines, they are interested in true multimedia content combining different media assets into multimedia documents like PowerPoint presentations and Flash files. In a user study with 20 participants, the authors conducted a formative evaluation based on the think-aloud method and semi-structured interviews in order to obtain requirements to a future web search engine for multimedia content. The interviews are complemented by a paper-and-pencil questionnaire to obtain quantitative information. As a result, the authors elicit requirements to a web search engine for multimedia content. Among them, scalability and personalization of the presented information are identified as the main goals. Based on the requirements, they present mockups demonstrating the user interface of a future multimedia search and retrieval engine.
Article Preview

1. Introduction

Multimedia content, which is provided by PowerPoint presentations or Flash documents, is widely adopted and can be found in any domain. Despite the growing interest in multimedia web search, most major web search engines currently offer only limited multimedia search functionality. Unlike existing image or video search engines, the content a multimedia search engine has to deal with is the coherent combination of different types of media assets such as images, videos, audios, and text. Those media assets can be animated or rearranged by interactions (Candan & Sapino, 2010). The semantic within multimedia documents is given through relations dealing with time and space of media assets (Boll, Sandhaus, Scherp, & Westermann, 2007), e.g., duration of a video, width, height, and position of an image, but these are just partially taken into account by previous systems. Research on search engines dealing with a single media type like images and videos is well established and there exist studies that investigate how users apply media search engines (Maniu, O'Hare, Aiello, Chiarandini, & Jaimes, 2013; Kofler & Lux, 2009). However, regarding search and retrieval of true multimedia there is a gap in the research. Thus, we have developed an early prototype of a search engine for multimedia content on the web (Tingvold, Stohr, Schneider, & Amundsen, 2013)1 (cf. Figure 1). The prototype offers users to search and explore for multimedia content. The goal of this early search engine was to provide users an initial idea of what a retrieval system for true multimedia could look like and to use this system to bootstrap requirement elicitation and detailed understanding of users’ needs for semantic multimedia search. On the basis of this prototype, we conducted a user study evaluating the features for a future multimedia search engine. The current prototype is developed for keyword-based queries only. There are filters for audio, video, animation and interaction. These filters are applied by logical OR operators. The preview of the multimedia documents in the result list indicates if the document contains media assets fulfilling the filter. The thumbnails, which show the first frame or the first slide of the multimedia document, are of different sizes, starting with larger ones in the upper position to smaller ones in the lower position of the results page. Hovering over a thumbnail shows an animated preview of the document. Clicking on a document opens a detail view in the lower part of the search engine’s window.

Figure 1.

Fulgeo search interface and result view pagination is used and limits the result list to 20 documents per page. Overall, the prototype has about 4000 multimedia documents in the database.


In the subsequent sections, we present the related work in multimedia search and studies on media search engines. In Sections 3 and 4, we describe the conducted user study and its results. Based on these results, we derive requirements to a future multimedia search engine in Section 5. We designed mockups combining good features of the existing prototype, with new requirements collected from the survey, and further ideas derived by the interview section (Section 6). Finally, we discuss the overall results and the limitations of our study (Sections 7 and 8).

Complete Article List

Search this Journal:
Volume 15: 1 Issue (2024): Forthcoming, Available for Pre-Order
Volume 14: 1 Issue (2023)
Volume 13: 4 Issues (2022): 1 Released, 3 Forthcoming
Volume 12: 4 Issues (2021)
Volume 11: 4 Issues (2020)
Volume 10: 4 Issues (2019)
Volume 9: 4 Issues (2018)
Volume 8: 4 Issues (2017)
Volume 7: 4 Issues (2016)
Volume 6: 4 Issues (2015)
Volume 5: 4 Issues (2014)
Volume 4: 4 Issues (2013)
Volume 3: 4 Issues (2012)
Volume 2: 4 Issues (2011)
Volume 1: 4 Issues (2010)
View Complete Journal Contents Listing