Evidencing Quality: Using the Sloan-C Quality Scorecard

Evidencing Quality: Using the Sloan-C Quality Scorecard

Kaye Shelton (Lamar University, USA), Karen L. Pedersen (Northern Arizona University, USA) and Lisa A. Holstrom (Academic Partnerships, USA)
DOI: 10.4018/978-1-4666-5051-0.ch020


In an era of tightening institutional budgets, ever increasing online enrollments and greater calls for accountability from multiple directions, the need for online program administrators to continually assess the quality of their overall operations has never been greater. But even vigilant administrators have had difficulty defining “quality” and were unable to benchmark to other programs, as standards for quality varied. To address this daunting task, a research-based standard assessment tool known as the Quality Scorecard for the Administration of Online Education Programs was developed. The primary goals of this case are to Provide a brief overview of the scorecard highlighting the key areas of the quality indicators and the scoring protocol and present practical applications for this research-based assessment tool as evidenced through the administration of the scorecard by three online administrators at four institutions. By focusing on the ways in which different administrators at diverse institutions (public and private, large and small) used the Quality Scorecard to benchmark their online operations against a standard, we are able to illustrate how continuous improvement opportunities, impacting on both learning effectiveness and program improvement, can be implemented at the program or institutional level.
Chapter Preview

Setting The Stage

What is the Quality Scorecard?

In an effort to fill a gap in the literature and provide a practical tool for the evaluation of quality of online education programs in higher education, The Quality Scorecard for the Administration of Online Education Programs was developed through a systematic research process in 2010. Adopted by the Sloan Consortium (Sloan-C), an organization dedicated to quality online education, as their recommended framework for program assessment, the Sloan-C Quality Scorecard offers a process for determining areas of quality and improvement needs from an administrator’s perspective and may also be used for strategic planning and online program development. The Sloan-C Quality Scorecard has also been used as a tool to support accreditation requirements.

The Sloan-C Quality Scorecard was the result of a six-month Delphi Study conducted by Dr. Kaye Shelton using experts throughout the U.S. The Delphi method of research, developed by the Rand Corporation in the early 1950s by Norman Dalkey and Olaf Helmer (1963), was used to gain statistical consensus from experts in online education administration as “it replaces direct confrontation and debate by a carefully planned, anonymous, orderly program of sequential individual interrogations usually conducted by questionnaires” (Brown, Cochran, & Dalkey, 1969, p. 1). In fact, the Delphi method is often used for consensus on topics or decisions considered to be subjective and that usually do not have a single correct solution. It also prevents groupthink since the topic experts are never together at one time, and are usually anonymous.

Because the strength of a Delphi study is related to the qualifications of the experts selected to participate (Rossman & Eldredge, 1982), the sampling frame of online education administrators in higher education was identified by Sloan-C through a gatekeeper process. For the study, 43 panel members participated out of the 76 invited experts. Table 1 shows the institutional classification for the panel members. More than 83% of the panel members had nine or more years of experience in the administration of online education programs. Of the 43 panel members, 56% were from large public institutions.

Table 1.
Institutional classification for expert panel members who participated
Institutional ClassificationTypeSizeTotal
Public (4 year)Non-profitLarge24
Public Community College (2 year)Non-profitLarge2
Private (4 year)Non-profitLarge4
Private (4 year)For-profitLarge1
Private Faith-Based (4 year)Non-profitLarge1
Public (4 year)Non-profitMedium2
Private (4 year)Non-profitMedium3
Private Faith-based (4 year)Non-profitMedium3
Public (4 year)Non-profitSmall1
Private (4 year)Non-profitSmall2

The Sloan-C Quality Scorecard may be used to demonstrate the overall quality of individual online education programs or an institution, no matter what size or type. There are nine categories with a number of quality indicator statements in each category of the Scorecard:

  • Institutional Support (4 indicators.)

  • Technology Support (6 indicators.)

  • Course Development and Instructional Design (12 indicators.)

  • Course Structure (8 indicators.)

  • Teaching and Learning (5 indicators.)

  • Social and Student Engagement (1 indicator.)

  • Faculty Support (6 indicators.)

  • Student Support (17 indicators.)

  • Evaluation and Assessment (11 indicators.)

The following steps of implementation are recommended:

  • 1.

    Appropriate personnel should examine the online program for evidence of each of the 70 quality indicators. Based upon the level of evidence observed, one of the following values is selected: 0 points - not observed, 1 point - insufficient, 2 points - moderate use, 3 points - completely meets criteria.

  • 2.

    For each indicator, and score, examples of observed evidence and written justifications should be provided. For example, the first indicator listed in the Institutional Support category states: The institution has put in place a governance structure to enable effective and comprehensive decision making related to distance learning. To substantiate the score for this indicator, the evidence could be documents such as digital copies of organizational charts, reporting structures, and advisory committee minutes demonstrating how a decision is processed.

The score for each indicator is totaled which determines the overall level of program quality observed:

  • A perfect score = 210 points.

  • 90-99% = 189-209 - Exemplary (little improvement is needed)

  • 80-89% = 168-188 - Acceptable (some improvement is recommended)

  • 70-79% = 147-167 - Marginal (significant improvement is needed in multiple areas)

  • 60-69% = 126-146 - Inadequate (many areas of improvement are needed throughout the program)

  • 59% and below = 125 points and below - Unacceptable.

The quality scorecard is available on the Sloan Consortium (Sloan-C) Website at http://sloanconsortium.org/quality_scoreboard_online_program. While anyone may use the Quality Scorecard process, for Sloan-C institutional members, an interactive tool is provided online that collects artifacts of evidence and written justifications for scoring, which in turn, becomes a self-study of the program.

Complete Chapter List

Search this Book: