What is the Quality Scorecard?
In an effort to fill a gap in the literature and provide a practical tool for the evaluation of quality of online education programs in higher education, The Quality Scorecard for the Administration of Online Education Programs was developed through a systematic research process in 2010. Adopted by the Sloan Consortium (Sloan-C), an organization dedicated to quality online education, as their recommended framework for program assessment, the Sloan-C Quality Scorecard offers a process for determining areas of quality and improvement needs from an administrator’s perspective and may also be used for strategic planning and online program development. The Sloan-C Quality Scorecard has also been used as a tool to support accreditation requirements.
The Sloan-C Quality Scorecard was the result of a six-month Delphi Study conducted by Dr. Kaye Shelton using experts throughout the U.S. The Delphi method of research, developed by the Rand Corporation in the early 1950s by Norman Dalkey and Olaf Helmer (1963), was used to gain statistical consensus from experts in online education administration as “it replaces direct confrontation and debate by a carefully planned, anonymous, orderly program of sequential individual interrogations usually conducted by questionnaires” (Brown, Cochran, & Dalkey, 1969, p. 1). In fact, the Delphi method is often used for consensus on topics or decisions considered to be subjective and that usually do not have a single correct solution. It also prevents groupthink since the topic experts are never together at one time, and are usually anonymous.
Because the strength of a Delphi study is related to the qualifications of the experts selected to participate (Rossman & Eldredge, 1982), the sampling frame of online education administrators in higher education was identified by Sloan-C through a gatekeeper process. For the study, 43 panel members participated out of the 76 invited experts. Table 1 shows the institutional classification for the panel members. More than 83% of the panel members had nine or more years of experience in the administration of online education programs. Of the 43 panel members, 56% were from large public institutions.
Table 1. Institutional classification for expert panel members who participated
| Institutional Classification | Type | Size | Total |
| Public (4 year) | Non-profit | Large | 24 |
| Public Community College (2 year) | Non-profit | Large | 2 |
| Private (4 year) | Non-profit | Large | 4 |
| Private (4 year) | For-profit | Large | 1 |
| Private Faith-Based (4 year) | Non-profit | Large | 1 |
| Public (4 year) | Non-profit | Medium | 2 |
| Private (4 year) | Non-profit | Medium | 3 |
| Private Faith-based (4 year) | Non-profit | Medium | 3 |
| Public (4 year) | Non-profit | Small | 1 |
| Private (4 year) | Non-profit | Small | 2 |
The Sloan-C Quality Scorecard may be used to demonstrate the overall quality of individual online education programs or an institution, no matter what size or type. There are nine categories with a number of quality indicator statements in each category of the Scorecard:
- •
Institutional Support (4 indicators.)
- •
Technology Support (6 indicators.)
- •
Course Development and Instructional Design (12 indicators.)
- •
Course Structure (8 indicators.)
- •
Teaching and Learning (5 indicators.)
- •
Social and Student Engagement (1 indicator.)
- •
Faculty Support (6 indicators.)
- •
Student Support (17 indicators.)
- •
Evaluation and Assessment (11 indicators.)
The following steps of implementation are recommended:
- 1.
Appropriate personnel should examine the online program for evidence of each of the 70 quality indicators. Based upon the level of evidence observed, one of the following values is selected: 0 points - not observed, 1 point - insufficient, 2 points - moderate use, 3 points - completely meets criteria.
- 2.
For each indicator, and score, examples of observed evidence and written justifications should be provided. For example, the first indicator listed in the Institutional Support category states: The institution has put in place a governance structure to enable effective and comprehensive decision making related to distance learning. To substantiate the score for this indicator, the evidence could be documents such as digital copies of organizational charts, reporting structures, and advisory committee minutes demonstrating how a decision is processed.
The score for each indicator is totaled which determines the overall level of program quality observed:
- •
A perfect score = 210 points.
- •
90-99% = 189-209 - Exemplary (little improvement is needed)
- •
80-89% = 168-188 - Acceptable (some improvement is recommended)
- •
70-79% = 147-167 - Marginal (significant improvement is needed in multiple areas)
- •
60-69% = 126-146 - Inadequate (many areas of improvement are needed throughout the program)
- •
59% and below = 125 points and below - Unacceptable.
The quality scorecard is available on the Sloan Consortium (Sloan-C) Website at http://sloanconsortium.org/quality_scoreboard_online_program. While anyone may use the Quality Scorecard process, for Sloan-C institutional members, an interactive tool is provided online that collects artifacts of evidence and written justifications for scoring, which in turn, becomes a self-study of the program.