Designing, Deploying, and Evaluating Open-Access Online Delphi Studies (OAODS)

Introduction

Classic Delphi studies involve obtaining forecasts about the future by seating selected certified domain experts to go through one or more rounds of questions. Rounds, or iterations, generally continue until the expert group arrives at a consensus or stability. Initial topics engaged were those dealing with sensitive military research. In general, Delphi Methods (DM) [also referred to as “Delphi Exercises” (Turoff & Hiltz, 1996, p. 2)] are applied to complex problems that are “not compatible to linear or precise analytical techniques, and where subjective judgement on a collective basis could illuminate new perspectives” (Donohoe, Stellefson, & Tennant, 2012, p. 40). Ill-structured problems are those with “unclear goals and incomplete information” (Voss & Post, 1988). These challenges may be so nascent and new that there may be no defined paths forward, as-yet. The Delphi process enables experts to consider “what could/should be” (Miller 2006), so imagination combined with expertise can be relevant.

Early versions of Delphi studies in the late 1940s and 1950s apparently occurred in face-to-face contexts, and over time, online methods (electronic Delphi methods) have been harnessed as well. This work introduces a particular type of electronic Delphi study, dubbed the Open-Access Online Delphi Studies (OAODS). OAODS instruments may be created on and delivered through online survey platforms, dedicated specifically designed platforms for e-Delphi studies, and even learning management systems (LMSes), based on functionalities. A variety of technologies that may be harnessed for this process (Aengenheyster, et al., 2015), and over the years, a number of Internet-based research tools have gained traction. There has been work on building dedicated technology systems to enable some of these processes (Gomez-Sanz & Fernandez, 2015). Of interest, various types of socio-technical systems have been derived from Delphi methods and “emerged under names such as prediction markets, collaborative tagging, recommender systems…and social networks that usually serve a commercial objective” (Linstone & Turoff, 2011, p. 1713). Other areas that use ideas from the Delphi method include “Decision Support Systems (GDSS), Collaborative Systems, and Collective Intelligence, and CMC” (computer mediated communications) (Linstone & Turoff, 2011, p. 1713). Delphi methods have been used for non-research purposes as well, such as to “scaffold complex CMC (author note: computer-mediated communications) discussions by large groups of students learning together online” (Turoff, Hiltz, Yao, Li, Wang, & Cho, 2006, p. 76). A Delphi approach was used to create a multi-attribute decision analysis model, with weighting of the utility factors (Chan, Yung, Lam, Tam, & Cheung, 2001).

Based on integrations with social media platforms (social networking sites, microblogging sites, blogging sites, news-sharing sites, video sharing platforms, image sharing platforms, publication platforms, and others), open calls may be made seamlessly to a broad potential audience. [For example, Qualtrics has integrations with publishing platforms like WordPress® (Benton, Pappas, & Pappas, 2011) as well as social media ties with Twitter®, Facebook®, Reddit®, LinkedIn®, Google Plus®, Pinterest®, and others.] Micro-payment work platforms enable the seating of panels to respond to designed surveys, as do online research platforms. Open-links themselves may be broadcast via electronic mailing lists, websites, person-to-person emails, text messages, and other methods; they may be conveyed as texts, images, two-dimensional bar codes/Quick Response (QR) codes, and other forms.