Abstract
In this chapter, the authors describe their guiding principles for development and improvement of an online program, offering specific examples of how these have played out. They then describe how an evaluative thinking approach allowed them to build an environment conducive to creative design and continual innovations. Specifically, they have a habit of meeting to focus on critical review of work, they value and continually incorporate feedback from diverse stakeholders, and they challenge assumptions through consistent consideration of data. They provide examples throughout the chapter of how an evaluative thinking approach has looked in practice. By including concrete examples, this chapter provides insights to those planning a new program or refining one in place, recognizing that evaluative thinking is a mindset that is valuable in many contexts.
TopBackground
Even before the COVID-19 pandemic accelerated the shift to online learning, online education was already on the rise in the United States. Recognizing this trend, the [redacted] offered an internal eLearning Innovation Initiative (eLII) funding program in 2015. This grant program coincided with a time when policymakers were emphasizing the significance of evidence-based design and evaluation in publicly funded programs. Faculty in [redacted] recognized an opportunity to meet the increasing demand for professionals skilled in academic and applied research by expanding graduate studies, advancing research or evaluation work, and extending the program to students who could not physically attend the [redacted]. Thus, a team of [redacted] faculty, under the leadership of [redacted], submitted a proposal and received grant funding to establish an online master's program in Research Methods in Education.
[redacted] developed the Research Methods in Education program to offer a versatile skill set in research methods, applicable across disciplines, to address the evolving needs of students and employers in the dynamic job market. Its expansion into an online modality was designed to enhance accessibility for students from diverse backgrounds. From its inception to current practice, faculty members with expertise in research methodologies have played a crucial role in course design and delivery. The program not only supports professional growth but also acts as a catalyst for new career paths and admission to Ph.D. programs. Additionally, it equips students in teaching and research roles with essential skills.
The program grew organically based on the aspirations of students seeking post-graduation positions and pursuing courses to acquire the necessary credentials. The curriculum development focused on constructing a comprehensive master's program that expanded to include the Graduate Certificate in Research Methods, thereby broadening the program's scope and reach. Valuable feedback from employers and recognized demands in fields such as testing, medicine and healthcare, institutional research, and evaluation across the nation informed the program's design. Offering core courses online in an asynchronous format enhanced the program's marketability, providing flexibility and fostering collaborations across departments and colleges within the university.
The expansion into online education, making courses and degrees more accessible to a wider range of students, was arguably one of the most significant features of the program's development. Moreover, this expansion provided an opportunity for faculty members to showcase their expertise in research methodologies. The program also played a crucial role in advancing the professional growth of its students, enhancing their careers by opening new paths or securing promotions. The comprehensive skill set offered also made its graduates highly competitive in admissions to Ph.D. programs nationwide. Furthermore, the program supported the growth of students as stronger teaching assistants and research assistants, equipping them with the necessary tools to contribute effectively to the department's endeavors. Ultimately, the program emerged from the intersection of student needs, industry demands, and faculty expertise. Its impact extended beyond the department, offering valuable research skills that were marketable across disciplines and providing opportunities for professional and academic advancement and collaborative partnerships.
Key Terms in this Chapter
Student Learning Outcomes: Specific statements that describe the knowledge, skills, and abilities that students are expected to acquire or demonstrate as a result of their educational experiences. They are typically measurable and provide a framework for assessing student progress and program effectiveness.
Inclusive online community: A virtual learning environment or online platform that fosters a sense of belonging, participation, and respect for all individuals, regardless of their background, identity, or abilities. It promotes collaboration, interaction, and equal opportunities for engagement and learning among all participants.
Academic Rigor: The high standards, intellectual challenge, and depth of learning expected in an educational setting. It involves engaging students in critical thinking, problem-solving, and analytical skills to ensure a thorough understanding of the subject matter.
Plan-Do-Study-Act (PDSA) cycle: A systematic framework for continuous improvement that involves planning a change or intervention, implementing it, studying the effects or outcomes, and then acting on what has been learned to refine or modify the approach.
Stakeholder: An individual, group, or organization that has a vested interest or concern in a particular project, initiative, or organization. In an educational context, stakeholders can include students, parents, teachers, administrators, policymakers, community members, and other relevant parties who are impacted by or have influence over educational decisions and outcomes.
Evaluative thinking: The ability to critically analyze and assess information, situations, and outcomes objectively and reflectively. It involves using evaluative skills, such as collecting and analyzing data, considering multiple perspectives, and making informed judgments to improve practices and decision-making.
Learning Management System (LMS): A software application or platform used by educational institutions and organizations to manage, deliver, and track various aspects of the learning process. It typically includes features for content management, course administration, communication, assessment, and reporting.
Curriculum Mapping: The process of organizing and aligning the components of a curriculum, such as learning objectives, content, instructional strategies, and assessments, to ensure a coherent and comprehensive educational experience. It helps educators identify gaps, redundancies, and opportunities for improvement in the curriculum.
Reflective Practice: The deliberate and thoughtful process of examining one's thoughts, actions, and experiences to gain insight and improve professional or personal development. It involves analyzing and evaluating one's own strengths, weaknesses, and areas for growth, and using this self-reflection to inform future decision-making and actions.
Substantive interaction: Meaningful and substantial engagement and communication between individuals within an educational context. It involves active participation, exchange of ideas, and dialogue that promotes deep learning, critical thinking, and collaboration. Substantive interaction can occur between students, between students and instructors, or in group discussions and activities.
Modular Design: An approach in curriculum or instructional design where learning materials and content are organized into discrete, self-contained modules or units. These modules can be rearranged, combined, or customized to accommodate different learning needs, preferences, and contexts.
Self-Directed Learning: An educational approach where individuals take responsibility for their own learning process. It involves learners actively identifying their learning goals, determining the necessary resources and strategies, and monitoring their progress. Self-directed learners are motivated, independent, and capable of self-regulation in acquiring knowledge and skills.
Universal design principles: A set of guidelines and strategies aimed at creating products, environments, and educational experiences that are accessible and usable by the widest range of individuals, including those with disabilities. The principles emphasize inclusivity, flexibility, and designing for diverse needs and preferences, ensuring that all learners can fully engage and benefit from educational opportunities.
Data-Driven: An approach or decision-making process that relies on the analysis and interpretation of relevant data and information. In an educational context, being data-driven involves using data from various sources, such as assessments, evaluations, and research, to inform instructional strategies, curriculum design, and policy decisions.
Accessibility: The design and provision of educational materials, resources, and technologies that are usable and available to all individuals, including those with disabilities. It aims to eliminate barriers and ensure equal access to education for everyone, regardless of their physical or cognitive abilities.
Quality Matters: A faculty-centered, peer review process and set of standards for the design and delivery of online and blended courses. It provides a framework and guidelines to ensure the quality and effectiveness of online learning experiences, with a focus on learner engagement, alignment of learning objectives, assessment strategies, and course design.
Asynchronous: A type of learning or communication that does not require the participants to be present or engage in real-time interaction simultaneously. In an asynchronous environment, learners have the flexibility to access and participate in learning activities at their own pace and convenience.
Logic Model: A visual representation that outlines the logical connections between program resources/inputs, activities, outputs, outcomes, and impact. It helps to articulate and understand the theory of change underlying a program or intervention and facilitates planning, implementation, and evaluation.