Designing, Deploying, and Evaluating Open-Access Online Delphi Studies (OAODS)

Designing, Deploying, and Evaluating Open-Access Online Delphi Studies (OAODS)

DOI: 10.4018/978-1-5225-8563-3.ch003


This chapter describes the work of creating multimodal open-access online Delphi studies (OAODS). These are electronic Delphi studies that do not begin with an invited group of identified experts to seat a Delphi panel but rather with self-identified domain-specific authorities active on the Social Web, with post-data-collection vetting of the participants (when knowable) and their responses. This chapter explores how to design such instruments with efficacy and nuance, and built-in tests of respondent expertise, and fraud detection, and further, how to test such instruments for efficacy, reliability, and validity, while using some of the latest features available in online survey research platforms. The platform used in this work is the Qualtrics Research Suite.
Chapter Preview


Classic Delphi studies involve obtaining forecasts about the future by seating selected certified domain experts to go through one or more rounds of questions. Rounds, or iterations, generally continue until the expert group arrives at a consensus or stability. Initial topics engaged were those dealing with sensitive military research. In general, Delphi Methods (DM) [also referred to as “Delphi Exercises” (Turoff & Hiltz, 1996, p. 2)] are applied to complex problems that are “not compatible to linear or precise analytical techniques, and where subjective judgement on a collective basis could illuminate new perspectives” (Donohoe, Stellefson, & Tennant, 2012, p. 40). Ill-structured problems are those with “unclear goals and incomplete information” (Voss & Post, 1988). These challenges may be so nascent and new that there may be no defined paths forward, as-yet. The Delphi process enables experts to consider “what could/should be” (Miller 2006), so imagination combined with expertise can be relevant.

Early versions of Delphi studies in the late 1940s and 1950s apparently occurred in face-to-face contexts, and over time, online methods (electronic Delphi methods) have been harnessed as well. This work introduces a particular type of electronic Delphi study, dubbed the Open-Access Online Delphi Studies (OAODS). OAODS instruments may be created on and delivered through online survey platforms, dedicated specifically designed platforms for e-Delphi studies, and even learning management systems (LMSes), based on functionalities. A variety of technologies that may be harnessed for this process (Aengenheyster, et al., 2015), and over the years, a number of Internet-based research tools have gained traction. There has been work on building dedicated technology systems to enable some of these processes (Gomez-Sanz & Fernandez, 2015). Of interest, various types of socio-technical systems have been derived from Delphi methods and “emerged under names such as prediction markets, collaborative tagging, recommender systems…and social networks that usually serve a commercial objective” (Linstone & Turoff, 2011, p. 1713). Other areas that use ideas from the Delphi method include “Decision Support Systems (GDSS), Collaborative Systems, and Collective Intelligence, and CMC” (computer mediated communications) (Linstone & Turoff, 2011, p. 1713). Delphi methods have been used for non-research purposes as well, such as to “scaffold complex CMC (author note: computer-mediated communications) discussions by large groups of students learning together online” (Turoff, Hiltz, Yao, Li, Wang, & Cho, 2006, p. 76). A Delphi approach was used to create a multi-attribute decision analysis model, with weighting of the utility factors (Chan, Yung, Lam, Tam, & Cheung, 2001).

Based on integrations with social media platforms (social networking sites, microblogging sites, blogging sites, news-sharing sites, video sharing platforms, image sharing platforms, publication platforms, and others), open calls may be made seamlessly to a broad potential audience. [For example, Qualtrics has integrations with publishing platforms like WordPress® (Benton, Pappas, & Pappas, 2011) as well as social media ties with Twitter®, Facebook®, Reddit®, LinkedIn®, Google Plus®, Pinterest®, and others.] Micro-payment work platforms enable the seating of panels to respond to designed surveys, as do online research platforms. Open-links themselves may be broadcast via electronic mailing lists, websites, person-to-person emails, text messages, and other methods; they may be conveyed as texts, images, two-dimensional bar codes/Quick Response (QR) codes, and other forms.

Key Terms in this Chapter

Round: An iteration or “wave” of a Delphi study.

Dissensus: Dissent, disagreement.

Consensus: Consent, agreement.

Real-Time Delphi Study: An online Delphi study with potentially only one round.

Expert Panel: The individuals who have varying levels of expertise in a particular domain, mixed-domain, or peripheral domain.

Discussion Leader: Facilitator of a Delphi study.

Open-Access Online Delphi Studies (OAODS): Often-continuous online (electronic) Delphi studies that involve open links that self-professed experts may access and respond to (requiring validation of expertise).

E-Delphi Study: An electronic Delphi study deployed off of a socio-technical platform, such as a dedicated platform, an online survey platform, or social media platform.

Debriefing: An elicitation of information to better understand a phenomenon or event.

Delphi Study: A type of research involving experts, who engage in single- or multi-stage rounds of questions and other elicitations, in order to understand elusive future events or other topics.

Complete Chapter List

Search this Book: