Play or Vote: Matching Games as New Approach for Design Evaluation in Innovation Contests

Play or Vote: Matching Games as New Approach for Design Evaluation in Innovation Contests

Jörg B. A. Haller, Katja Hutter, Johann Füller, Kathrin M. Möslein
DOI: 10.4018/978-1-4666-0149-9.ch026
(Individual Chapters)
No Current Special Offers


This chapter displays IT-based innovation contests as new means to enrich a company’s “design-ideas” by the creativity of a multiplicity of external designers and enthused users all over the world. Further and foremost, it introduces the application of online evaluation games as method to elicit promising contributions in innovation contests. Two design-oriented innovation contests – “style your smart” and “discover Lattea” – serve as field experiments to explore the applicability and use of games for the evaluation of designs. Results indicate that online evaluation games help identifying the most promising designs in an innovation contest and overcoming some of the biases caused in typical evaluation settings.
Chapter Preview


Pushed through concepts like crowdsourcing (Howe, 2008; Kozinets et al., 2008), co-creation (Winsor, 2005), and open innovation (Chesbrough, 2003), firms increasingly use the creativity, skills, and intelligence of millions of individuals encountered on the Internet as source for innovative ideas. Building on the means of competition, innovation contests are one particular method to do so. Within the course of such contests, companies invite interested users to deal with a certain topic or product range, to show their talent by uploading their creative content, and further discuss and share their insights with like-minded people. Users can vote which idea or design they like best, discuss various topics by leaving comments at other users’ pin board, and compete for prizes (e.g. Morgan & Wang, 2010).

While such contests ensure a large variety of submissions, the identification of the best and most promising ones often causes large efforts. On the one hand, the mere magnitude of ideas generated can be overwhelming, on the other hand most approaches do not increase the chance of really selecting the best submissions nor reduce the risk of relying on the wrong ones (Bjelland & Wood, 2008; Riedl et al., 2010). The existence of social media and new information and communication technologies (ICT) offers new opportunities to encounter these challenges. In contrast to traditional competitions, many current innovation contests are based on IT-platforms. Participants can share their ideas, communicate with each other, establish relationships and even comment and evaluate others ideas. The latter is also referred to as open evaluation (Haller, 2011). Recent research indicates that open evaluation bears plenty of potential to support the selection of relevant submissions (Blohm et al., 2009; Möslein et al., 2010). Still, it is also recognized that many methods are prone to fraud. By using multiple accounts, participants can vote for themselves to increase their chance of winning or by voting down competitors, respectively. Hence, effective open evaluation has to avoid these pitfalls, while still tapping into the wisdom of the crowd.

Online games seem to be a promising approach. First experiments show the suitability of online games to elicit user preferences, while making it harder to cheat (Hacker & von Ahn, 2009). By predefining the ideas, designs, concepts or solutions to be judged, participants cannot vote for the submission they want to support. Further, games are perceived as an enjoyable and fun activity, leveraging the willingness of participants to share their knowledge. However, so far only a few companies use online games for this purpose. This paper introduces and discusses the use of games with a purpose for the evaluation and identification of most promising design ideas. Further, we investigate whether online evaluation games attract the attention of participants and if so, how their results correlate with other approaches focusing on the elicitation of participants’ preferences.

The remainder of this chapter is structured as follows. First, we provide an overview of innovation contests, followed by an overview on evaluation in innovation contests in general and of games with a purpose (GWAPs) as means of interest. Then we outline the methodical approach, before presenting the results of two studies and closing the paper with the discussion, future research directions and conclusion.

Key Terms in this Chapter

Open Evaluation: All types of evaluations that represent and bundle the judgment of people who are not part of the general group of decision makers.

Online Evaluation Game: Use of a matching game approach to evaluate ideas, designs or comparable items to identify user preferences.

Innovation Contest: (IT-based) competition of innovators who use their skills, experience, and creativity to provide a solution for a particular contest challenge defined by an organizer, which represent one of the most frequent manifestation of crowdsourcing for creative or complex tasks.

Games with a Purpose: “Games that are fun to play and at the same time collect useful data for tasks that computers cannot yet perform.” ( Hacker & von Ahn, 2009 : p. 2)

Crowdsourcing: Outsourcing of a formally internal conducted activity to a large and undefined or at least loosely defined crowd of people in form of an open call and, thus, to be distinguished from traditional outsourcing towards one or few contractors. Crowdsourcing can comprise of trivial, creative or complex tasks.

Open Innovation: Inbound and outbound activities of a company to optimize its innovation process, whereby inbound refers to the integration of external knowledge or ideas, while outbound encompasses activities like licensing of a companies’ patents.

Matching Game: A two-player online game, where randomly matched partners have to guess which of two pictures or comparable options their partner will prefer, and where both partners receive points if their guess matches.

Complete Chapter List

Search this Book: