Sample and Courses of Study
The current study included a sample of n=922 students. All students completed one of two possible course tracks for the data analytics concentration of the data analytics program in the College of Information Technology. Data covers a 1-year period and thus includes 2 terms for each student. All courses are online as part of a competency-based higher education institute, Western Governors University. Thus, all summative assessments are completed online and can be completed at any time in the course (once the student has exhausted and/or mastered application of the skills and competencies inherent in the content, which is done on their own time and at their own pace).
The assessments are also performance-based (PAs). PAs are graded individually by human raters and based on a rubric rather than a cut score. All PAs in this course series have just one task but with multiple criteria or “aspects” which correspond with specific skills associated with the overall competency as reflected in the rubric (Baryla, Shelley, & Trainor, 2012). For example, in the case of data analytics, one aspect may correspond with the ability to clean and organize data, another refers to the subsequent analysis performed, and others focus on proficient use of the software used to perform data collection, cleaning, and analysis. It is also worth noting that every PA task has a written communications aspect (e.g., interpretation of findings to a certain audience) due to its importance in the program. Therefore, task aspects serve as items in the current analysis. Task aspects are scored based on a rubric where 0 = non-competent, 1 = approaching competence, and 2 = competent. Thus, there is a possibility of 0-2 points per task aspect. More detailed information on each course assessment, including the rubric, is located in the Appendix A.
There are two course tracks that run in parallel in terms of competencies and skills covered by course material and the summative assessment. The first, short track (C740, C742) included 246 students. The second, long track (C740, C996 and C997) included 426 students. The first course, C740: Fundamentals of Data Analytics, is an introductory course that emphasizes proficient use of excel, particularly for data cleaning, organization, and statistical analyses such as regression. All students in the data analytic program take this course. The second course is either C742: Data Science Techniques, which covers both Python programming to collect data from websites and use of R for regression analysis of collected data (and is taken in the same term as C740), or two courses that cover Python programming (C996: Programming in Python) and Regression in R (C997: R for Data Analysis) separately and respectively. Moreover, C996 and C997 are taken in separate terms. All four courses emphasize one or more of the following steps in data manipulation and analysis: use and application of a computer software program to collect, organize, and analyze data using multiple regression, and subsequently interpret the results (as well as explain the rationale behind code). Thus, certain university-level skills such as verbal communication and critical thinking are measured in all assessment scores.