Hear from the Experts on Pre-Tests in Informal Massive Open Online Courses

DR. GLICK: Good afternoon, Maria. Thank you so much for making time for this interview. According to a recent report by EdSurge, many colleges failed to invest in building online programs in the pre COVID era, even as demand for them increased. Now that the pandemic suddenly made online learning essential, colleges find themselves more willing to use online course libraries like those made by Coursera and edX. As the pandemic leads more colleges to embrace MOOCs, higher attrition will have a detrimental impact on students and institutions alike.

In your chapter, you report the results of an experimental study that examined the effects of pre-testing on performance and persistence in MOOCs. What are the effects of pre-testing – with and without feedback – on learning outcomes? Specifically, do pre-tests lead to better learning outcomes among MOOC students?

DR. JANELLI: Thanks for inviting me to write about this, Danny. In the study you mentioned, we found four compelling and somewhat surprising results. First, among ALL of the users in a Coursera course, pre-tests and feedback did not affect learning outcomes. Moreover, the presence of pre-tests actually decreased student persistence and course completion. However, among students who did complete the course, those who took the pre-tests achieved higher learning outcomes than those who did not. Finally, those who took the pre-tests in all five modules had greater learning outcomes. The short answer to your question is that pre-tests lead to better learning outcomes, but only for students who complete the entire course.

DR. GLICK: What are the implications of all this for instructional designers and MOOC providers?

DR. JANELLI: The implications are dependent upon the overall course goals. If the course is a traditional higher-ed course (taught by an instructor, taken for-credit), then instructional designers should include pre-tests because they lead to greater learning outcomes. (In our study, feedback did not have an effect on this, but more research is needed to understand the utility of feedback in this area.)

If the course is a traditional MOOC (informal, self-guided, not-for-credit), then instructional designers should not include pre-tests because pre-tests increase attrition. But that is an overly simplistic approach. There are several additional considerations. We know that pre-tests were effective for people who completed the course, but that pre-tests also decreased course persistence and completion. We also know that, among those who submitted pre-tests, those who submitted all five pre-tests achieved higher learning outcomes than those who submitted just one or two. This becomes an opportunity for instructional designers to identify online learning strategies that encourage participants to both submit pre-tests and complete the MOOC. This could be achieved through gamification, micro-credentialing, or by incorporating pre-test scores into the overall course grade.

A solution that would benefit the most students is also worth considering. Instructional designers could create separate paths for a given MOOC: one version of a course with no pre-tests for those who don’t intend to complete the course, and another version of the course with pre-tests for those who do intend to complete the course. This would maximize persistence for students who aren’t interested in taking pre-tests and maximize learning for students who are.

DR. GLICK: Research examining student characteristics in MOOCs, found that age, personal interest, and usefulness to one’s career are positively associated with student engagement.

What other personality traits are associated with improved performance and persistence in MOOCs?

DR. JANELLI:I don’t have a conclusive answer to this question based upon my own research. However, my colleagues and I are currently conducting a study that seeks to understand the relationship between non-cognitive factors and MOOC performance. Specifically, we’re using the Big Five Inventory Personality Test to better understand the characteristics of the student population and the relationship between those characteristics and engagement, persistence, and learning.

DR. GLICK: Transitioning from in-person to remote learning can present challenges for both teachers and students. As a result, a plethora of professional development programs have arisen, to support educators with tools and strategies for successful transition to remote learning. However, these learning experiences add both time and effort demands to teachers’ already busy schedules. In addition, in-person teacher training programs are usually unable to provide ongoing daily guidance for teachers, identify teachers’ areas of strength and weakness, or provide personalized feedback.

Given the many challenges associated with in-person professional development programs, what new considerations should designers and providers of in-person and/or online teacher education programs be keeping in mind?

DR. JANELLI: This is a great question and I think that the pandemic has shed light on both the challenges and opportunities of delivering quality professional development online. I suspect that many organizations will decide to offer PD in both formats, at least for the time being. My research and work focus on online learning, so I will stick to that part of the question.

The challenges of online PD are probably obvious to most people because of the ways in which we’ve relied upon Zoom and other video meeting software during the past 14 months. These include “Zoom fatigue,” creating meaningful digital collaborative workspaces, navigating disruptions (barking dogs and ringing doorbells simply don’t happen in face-to-face workshops), and fostering genuine interactions as we all exist within the confines of our little video squares. Despite these shortcomings, virtual PD has benefits that providers can leverage.

One benefit that comes to mind is accessibility. Teachers who can’t attend in-person events due to time constraints may be able to participate in virtual events. If they can’t, virtual events can easily be recorded and a) viewed by additional participants afterward and b) repurposed by the provider for additional PD workshops or educational materials. Additionally, some educators may have differing physical abilities that make online events more accessible to them.

Another benefit is scale and, related, diversity. Unless it’s at a national conference, face-to-face events tend to be small and local. Hosting professional development workshops online allows the provider to reach more participants in more places. With online PD, 50 teachers can come together to learn about the Next Generation Science Standards. Some might be from New York, some from Idaho, and some from Alabama. This variety of regional, professional, and personal perspectives has the potential to benefit participants and providers alike.

A third benefit is cost. It is cheaper to host a PD event online, and those savings can be passed on to educators and their school districts.

When it comes to considerations to keep in mind, I can draw upon my work at the American Museum of Natural History in New York City. The Museum has offered online teacher professional development for several decades, and my colleagues and I have learned a lot during that time. We schedule workshops around teachers’ schedules (or host them asynchronously). We model instructional strategies and provide free resources that teachers can use in their own classrooms. We incorporate interactive elements into the workshops or courses. We provide dedicated spaces where teachers can ask questions of their instructional team.

However, the most important considerations, especially in these times, are compassion and flexibility. Everyone has struggled due to COVID-19, but teachers have been disproportionately affected. Making space to acknowledge that struggle is the very least we can do for our educators – whether in person, or online.

DR. GLICK: Maria, thank you so much for sharing your valuable insights!

Danny Glick is a Research Affiliate at the University of California, Irvine’s Online Learning Research Center where he explores ways to improve student persistence and performance in online courses using early warning systems and light-touch interventions. He is a former visiting scholar at the University of California, Irvine’s School of Education where he investigated the effects of blended learning on the achievement of low-income students. Dr. Glick is also the Director of Pedagogical Implementation at Edusuft, a subsidiary of ETS, where he leads a team of EdTech implementation specialists. For the past 20 years, he has helped ministries of education and higher education institutions in 35 countries to shift from traditional instruction to online learning. Dr. Glick holds a PhD in Learning Technologies and a Master’s degree in Curriculum & Instruction, and has presented and published on topics including early warning systems, targeted interventions, student persistence, and learning design.

Dr. Maria Janelli is the senior manager of online teacher education programs at the American Museum of Natural History. She has spent her career designing, implementing, and researching e-learning applications at some of New York City's most renowned cultural and educational institutions. She holds a Ph.D. in educational psychology from the City University of New York, an MA in Anthropology from Columbia University, and an MA in English Literature from Rutgers University. Her research interests include: educational technology, instructional design, digital assessment and feedback, and formal and informal online learning.