An observation Protocol for Scaffolding Community of Inquiry and Its exemplary Practices in Language MooCs

As a response to a call to investigate the fundamental aspects regarding educational theory, research, designing and teaching of language massive open online courses (MOOCs), this study first developed a community of inquiry (CoI) observation protocol to observe the existing teaching, social, and cognitive presences in language MOOCs and tested its reliability using g-theory analysis. The results showed that the developed observation protocol is reliable, as evidenced by the large proportion of variance attributed to variation across courses rather than across raters. A follow-up d-study suggested that 5 and 11 raters were enough to reach moderate and substantial reliability coefficients, respectively. The study also identified exemplary practices that reflected high-level CoI presences in language MOOCs. The result not only highlighted the need to conduct observational studies to disentangle the dynamic interchanges that occur in language MOOCs but also provided practical guidelines to language educators interested in designing and teaching their own MOOCs.

As for students' cognitive engagement, scholars have called for improvement to the promotion of students' active cognitive engagement, to avoid low-impact, one-way teaching (Nie & Hu, 2018;Zhao, 2015). In addition, students preferred learning tasks that gave them a sense of relatedness (Jitpaisarnwattana et al., 2022c). What's more, reading and listening skills were found to be much easier to transmit to students via MOOCs than other language skills such as writing were (Vorobyeva, 2018). Also, the completion rate may no longer be the only indicator of learning success (e.g., Friðriksdóttir, 2021b;Jitpaisarnwattana et al., 2022a;Mac Lochlainn et al., 2021). Rather, as Jitpaisarnwattana et al. (2022a) argued, success should only be defined by the learners, and LMOOC designers should recognize learners' various needs and design course contents and activities accordingly in order to meet their learning expectations.

Measuring CoI Qualitatively in the online eFL Context
The instrument most often used to capture CoI presences is the CoI survey developed by Arbaugh et al. (2008). Despite this instrument's popularity, it has two important drawbacks in applying it to L2 context. The first is that most recent L2-focused CoI publications (e.g., Assalahi, 2020;Smidt et al., 2021) have adopted Arbaugh et al.'s original 34-item CoI survey without revisions. The situation is further complicated by the fact that some indicators of CoI presence -for example, integration and resolution -do not have corresponding EFL elements. A second concern is that the instrument alone cannot readily capture nuances of the dynamics of real-world situations, and thus may hinder scholars' descriptions of the formation of online CoIs (Kaul et al., 2018), and fail to inform instructional approach in designing and delivering effective online courses (Fiock, 2020;Szeto, 2015).
The authors therefore argue that observation-based CoI studies -which produce rich information about their selected cases, and teachers in particular (Smit et al., 2017) -could help resolve this impasse. In addition, observation enables a deep and thorough understanding of the collected data, including about what works and what does not work when teaching a particular L2 (Garza et al., 2018). However, a recent literature review on the data sources used in MOOC studies showed that observation was the least used data source, especially when comparing to data sources such as log data, survey, and achievement data (Zhu et al., 2022). An example of using observation is Yang et al. s' (2019) study that used a Classroom Video Observation Framework to study instructional practices in a synchronous Chinese as a foreign language class. As well as the aforementioned strengths of using observation, their study revealed an important limitation in coding. That is, although the three presences included in their coding schema were the three elements in the CoI framework, the presences' subcomponents did not match those in the CoI framework. Therefore, it remains unknown how CoI could be best implemented in observing online language classrooms. In another study, Berry (2017) proposed an interview protocol based on the CoI framework and used it to analyze 50 hours of video clips from four online classrooms, as well as the focal classes' associated discussion posts. This yielded new patterns and codes that had not been identified previously in other onlinelearning populations (Shea & Bidjerano, 2009;Stodel et al., 2006), which further highlighted the need to develop a qualitative-research tool, guided by the CoI framework, to capture online teaching, activities, and student work.
Accordingly, and building upon the foregoing literature review, this study proposes to develop a reliable CoI observation protocol for learning experiences in LMOOCs. We will then use the developed protocol to identify some exemplary practices in LMOOCs, and summarize these practices for the benefit of instructors in such courses. Our research questions are: 1. What is the reliability of the developed CoI observation protocol? 2. How many raters are needed to maximize the protocol's reliability? 3. Based on the observation results, what practices in the sampled LMOOCs can be deemed exemplary?

Sample
The present study selected six English LMOOCs belonging to two popular MOOC learning platforms, Coursera and XuetangX (Shah, 2019), with three courses from each platform. To make them comparable, the authors confined the six courses within undergraduate-level academic English courses. In choosing their sample, they adopted an approach similar to that used by Luo and Ye (2021): at the time of data collection, the chosen courses (a) were all among the top three courses on their respective sites by popularity, and (b) had been available to the public for at least two years. Details of the aims and enrollment numbers of each of these six courses are presented in Table 1. The authors then chose the first two and the last two units in each course as analysis units. That is to say, in total, there were 24 units coming from six different popular LMOOCs chosen for further observation.

development of the observation Protocol
The authors conducted four rounds of developing the Community of Inquiry in LMOOC Courses (COILMC) observation protocol. The first round was broadly based on Arbaugh et al.'s (2008) instrument, but also relied on the three standards (i.e., observability, transparency, and merging redundant items) proposed by Smit et al. (2017) to revise all items. All items that did not meet the three standards were discussed and revised. The second round of the development of the observation protocol involved integrating findings from some of the most recent studies involving CoI measurement, from three perspectives: (a) their qualitative findings in general (Caskurlu et al., 2021;Fiock, 2020), (b) their findings regarding application of the CoI framework in the field of EFL teaching and learning (Ge et al., 2022;Mo & Lee, 2017;Yang et al., 2019), and (c) their findings about use of the CoI framework to analyze existing MOOC courses (Cohen & Holstein, 2018;Goshtasbpour et al., 2020;Kaul et al., 2018;O'Riordan et al., 2020;Zou et al., 2021). In the third round, to render the original four-point Likert scale from Arbaugh et al. (2008) easier to understand, the authors also developed rubrics for each point. 1-point and 2-point responses would be deemed negative verdicts on the observed dimension, and 3-point and 4-point ones, positive verdicts. Lastly, to further polish the COILMC instrument, the authors provided the protocol to a group of 14 EFL instructors and teaching assistants with online teaching experience (10 of them in LMOOCs) and asked for their advice. The authors revised the items accordingly based on the feedback of the instructors and teaching assistants. After this final round of revision, the COILMC observation tool consisted of 36 items. A detailed description of the revisions can be found in Appendix I. The COILMC protocol is presented in Appendix II.

Rater Training
This study's six raters were recruited research assistants who were interested in teaching EFL online, and had either done so before or learned EFL online for at least a year. The training period lasted approximately three months in spring 2021, during which they met once a week for one and a half hours. The training had three main stages. During the first, the raters gathered and discussed prior CoI-related studies. In the second stage of the training, the lead author chaired a 90-minute discussion with all raters about their understanding of the observation protocol, to ensure that everyone had similar interpretations of each item. Then, in the third and final training stage, all raters observed two similar learning units from two different MOOC courses, and compared their observation results for variations in their scoring and the intentions and CoI understandings that underlay such scoring. They gave special attention to the extreme scores, and thoroughly discussed inter-rater discrepancies until all of them were resolved. To limit the chances that observation sequence would affect the ultimate scoring results, no fixed sequence was used; rather, they were told only that their scoring had to be completed within eight weeks. The raters were reminded to consult the observation protocol, as many times as they needed to, to ensure that their scores were aligned with the manual. They were also asked to create written description comments for exemplary examples for each observed item when observing different units and courses.

data Analysis
In answering this study's first two research questions, the main statistical approach to computing reliability was generalizability theory (g-theory), which holds that researchers must always consider multiple sources of measurement errors, which are termed "facets" (Shavelson & Webb, 1991). It is especially helpful when trying to disentangle issues related to measurement design (Hill et al., 2012). The facets in this study consist of units (u), courses (c) and raters (r). The authors followed a r*(u:c) design, in which u are nested within c and crossed with r. After identifying the g-theory design for the current study, the authors proceeded with two separate studies, namely a generalizability study (g-study) and a decision study (d-study) (Huebner & Lucht, 2019), of which the former decomposes the variance of the components, while the latter allows identification of the most cost-effective numbers of each facet, to maximize reliability (Huebner & Lucht, 2019). Both the g-study and the d-study were performed in R using the gtheory package. To answer the third research question regarding exemplary practices in LMOOCs, the authors first picked those courses with the CoI presence mean scores larger than three as the exemplary courses, and then collected raters' written description comments for the exemplary practices when observing those courses. All comments were coded and analyzed using a bottom-up scheme (Miles & Huberman, 1994) under the different aspect of each CoI presence. The lead author color-coded all documents and summarized indicators that emerged from the exemplary practices. Then, the second author reexamined the coding results, and discussed with the lead author until the final indicators were agreed upon. Table 2 sets forth the results of the g-study, in which four units were nested in six MOOCs and crossed with the six raters, regarding observed teaching presence, social presence, and cognitive presence. The variance in teaching presence that was attributable to MOOC was 78.1%, suggesting that the developed observational tool is quite sensitive to the kind of course that is being observed. The variance in the same aspect of CoI that was attributable the raters was just 2.7%, suggesting that they had been quite consistent in observing teaching presence based on the rubrics provided. The variance associated with the nesting effect was 1.0%, showing that the units nested within each course were quite consistent in terms of their teaching presence. There was also a 15.5% variance ascribable to the interaction between the course and the raters, but an F test showed that it was not significant. The g-study results for social presence exhibited a similar pattern. Of the variance in the observed social presence score, 85.2% was explained by courses, 2.2% by the six raters, 1.3% by the nesting effects, 5.4% by the interaction effect, and the remaining 6% by the residuals.

Reliability of the CoILMC observation Tool
Lastly, 58.4% of the variance in cognitive presence was explained by courses, followed distantly by the 24.1% accounted for by the interaction between course and raters (which again was found to be non-significant via F testing). The percentage of variance from the nesting effect was close to zero: i.e., all four units within a given course tended to show similar observation scores for this type of CoI presence. The rater effect made up 8.1% of the variance, and the residuals about 9.3%.
In sum, the reliability of the instrument across the three presences exhibited ideal reliability. That is, the authors were expecting that the variance associated with the course facet would be very high, and with the other facets, very low. In other words, the observed variations were reflections of variation across courses, rather than of variation across different raters.

Number of Raters Needed to Maximize Reliability
Next, the authors used the g-study results to establish the number of raters needed for different levels of reliability (Table 3). In the case of teaching presence, two raters were required for a moderate reliability of .60 to .80, and four, for a substantial reliability, i.e., larger than .80 (Shrout, 1998). For cognitive presence, the parallel numbers were two and five. For social presence, on the other hand, considerably more raters were needed: five and 11 for moderate and substantial reliability, respectively. Thus, five raters were sufficient if the goal was to reach a moderate level of reliability for the observation protocol, and 11 raters were sufficient if it was to reach a substantial level of reliability. Table 4 shows the mean score of the CoI presence in each course. Exemplary practices were then selected from the course with positive verdicts (i.e., scores larger than three) in each CoI presence. The exemplary teaching practices under the teaching presence is displayed in Table 5. The authors divided the teaching practices according to the three different dimensions of the teaching presence. After synthesizing the exemplary practices, the authors found that many of them were related to the facilitation behavior of teaching. The authors then classified it into four different indicators (Identifying areas of agreement and disagreement, Helping students to clarify their thinking, Keeping members engaged, and Encouraging students to explore new ideas). Identifying areas of agreement and disagreement 1) Telling the students to assess writing with a critical mind, and that there is not just one right way to do so 2) Using roundtable discussion sessions to present different views, e.g., when discussing the topic of identity, including multiple persons, each with different stories, backgrounds, and viewpoints 3) Making comparisons between good and bad writing

exemplary Practices in observed Courses
Helping students to clarify their thinking 1) Summarizing existing problems, e.g., in one case, the four major challenges to Chinese EFL students seeking to publish English papers 2) Presenting examples 3) Establishing knowledge trees, flow charts or other visualizations to show the interrelations among different topics 4) Providing peer-review opportunities 5) Providing explicit "Future directions" sessions Keeping members engaged 1) Conducting polls 2) Using diverse avatars (e.g., to represent students with different cultural and/or knowledge backgrounds) 3) Posing step-by-step questions in videos 4) Setting up roundtable discussion sessions 5) Recording videos in real campus settings 6) Hosting prestigious guest speakers Encouraging students to explore new ideas 1) Telling them explicitly to explore new concepts in their own time (e.g., after introducing the concept of lexical density and strategies in class, one instructor encouraged students to think of additional strategies after class) 2) Guiding them to explore new concepts implicitly, through setting up questions after each reading assignment 3) Embedding unfamiliar terms (e.g., run-on sentences, dangling structure, faulty parallelism) in discussions of texts, and encouraging students to further explore those concepts Adapting the course for students with different ability levels The exemplary teaching practices under the social presence is displayed in Table 6. Compared with the large amount of exemplary practices observed in teaching presence, all raters expressed that it was difficult for the students to perceive classmates as real. This was mainly due to the limited opportunities the focal MOOCs provided for collaboration and discussion. Furthermore, the overall level of group cohesion the raters observed was relatively low, probably due to the limited chances to communicate that the focal MOOCs afforded.

Aspect
Indicator Exemplary Practices

Direct instruction
Providing clear and precise instruction 1) Showing the core concepts in a clear and accurate way in slides 2) Providing both Chinese and English explanations in interpreting difficult points (e.g., when explaining some common mistakes made by Chinese EFL learners, one instructor not only pointed out the dangling structure, but also provided alternative explanations in Chinese) 3) Pausing videos and presenting short pop-up quizzes to facilitate understanding 4) using real writing samples from students of different levels 5) providing high-quality feedback in discussion forums in a timely manner, i.e., within not more than three days Table 6. Exemplary teaching practices under social presence

Aspect Indicator Exemplary Practices
Affective expression Providing opportunities for learners to know each other 1) Setting up "Meet your classmates" sections in their discussion forums 2) Including posts such as "Finding a language learning partner" that were published by learners spontaneously 3) Assignments that require collaboration were a further channel whereby students could become acquainted Seeing the instructor as real 1) Did not read the slides mechanically, but explained the content in a complementary way 2) Exemplified content using personal experience (e.g., one instructor frequently talked about her experience as a member of her department's hiring committee, to give students a deeper understanding of what good CVs and personal statements look like) 3) Added their personal thoughts to explanations of learning content 4) talked in a manner that suggested they and the students were sitting together in the same room 5) Taught from their real office.
Seeing classmates as real 1) Self-introduction sessions 2) Submission of assignments to discussion forums. For example, one observation comment said, "I could feel my classmates were real when I checked the writing assignments that they submitted." Table 6 continued on next page The exemplary teaching practices under the cognitive presence are displayed in Table 7. We found more exemplary practices were related to the first three stages of cognitive presence (i.e., triggering events, exploration, and integration), while resolution seems to show limited exemplary practices.

Aspect Indicator Exemplary Practices
Open communication Encouraging students to express their ideas "We encourage you to engage in conversation and collaboration in this course, because they are critical components of learning in MOOCs." Providing a large number of opportunities for communication per unit In one course, although its first unit did not cover very many content-related topics, the instructor still set up two discussion sections simply to facilitate student-student communication

Group cohesion
Seeking the learners' trust through describing the reputation of the course, the instructor, and/or the institution that the instructor belonged to "After all, this is a course/professor from ** university." Being acknowledged by the other course participants 1) Someone started a post regarding the use of "we" vs. "I" in a discussion forum. After that, he received 10 high-quality replies from other students discussing this issue in light of their own experiences 2) When another student questioned a part of a lecture about cutting too many words from a sentence, eight students expressed their confusion, and engaged in a heated discussion regarding how many words should be removed from sentences The addition of learning materials to enable further learning 1) In a unit on using active voice in writing, the instructor provided an article as a writing sample, and explicitly stated that learners were welcome to use it as a sample of exemplary writing style in their own time 2) Providing further learning materials that were not merely abundant, but also all relevant Appreciating a variety of perspectives -e.g., instructor, author, reviewer, and student Allowing students to understand how such perspectives could lead to different ways of writing or editing

Integration
Integrating new information to help answer questions that were raised When teaching about how to write scientific news stories, one teacher compared traditional articles against innovative ones, and then introduced the concept of the nut graf to explain the differences.

Comments
Teachers' comments on writing, which were important to learners' creation of solutions to academic-writing problems Encouraging learners to write summaries Using "do" and "don't" to summarize writing strategies; listing 10 common language errors to summarize learning materials; and establishing a writing routine that helps to construct solutions Resolution Explicitly encouraging students to use the knowledge they were acquiring in their MOOCs in other settings "Please spread the word and pass on the skills you've learned in this class to other scientists to help them become better communicators." The learning content itself Cover letters, grant proposals, as well as its assignment tend to be of great value in students' future lives

The development of CoILMC
The findings of this study extend the use of CoI from a widely adopted conceptual framework for understanding online teaching to teaching languages via MOOCs, and confirm the theoretical value of examining the construction of online language-learning communities using a CoI framework (Assalahi, 2020;Herrera Díaz & González Miy, 2017;Mo & Lee, 2017;Smidt et al., 2021;Sun et al., 2017). Specifically, adopting an analytical approach based on g-theory, this study tested the reliability of the developed LMOOC observation protocol -COILMC -and established the number of raters needed for observation using it to reach satisfactory reliability. Our developed and/or revised items in COILMC show that the three presences in the CoI framework were present in all six sampled LMOOCs, both as core elements and as overlaps among each of those three elements (Garrison et al., 2000). Second, the findings confirm the value of using a structured observation protocol to analyze the "dynamic interplay of various classroom processes and conditions" when studying L2 acquisition (Dörnyei, 2007, p. 178). A direct comparison of our protocol against the survey developed by Arbaugh et al. (2008) reveals four notable differences. The first is a shift from students' self-reported perspectives to raters' observational ones. The second is an expansion from items that focused on CoIs in discussion forums only, to include ones that focus on CoIs in all-around online course delivery (Caskurlu et al., 2021;Fiock, 2020), EFL teaching (Mo & Lee, 2017;Yang et al., 2019), and teaching in MOOC contexts (Cohen & Holstein, 2018;Goshtasbpour et al., 2020;Kaul et al., 2018;O'Riordan et al., 2020;Zou et al., 2021). The third is the addition of concrete examples from EFL that correspond to different dimensions within each CoI presence. Last but not least, the COILMC protocol was shown to be of high reliability. By using g-theory (Shavelson & Webb, 1991), the authors were able to examine the reliability of COILMC through partitioning the error variance into different sources of error. The results of the g-study suggested that the highest proportion of variance in observation scores was attributable to variations among the six observed courses. As well as being an obvious sign of the reliability of the instrument, this finding revealed that LMOOCs were indeed at quite different levels in terms of their formation of CoIs. On the one hand, this echoes Ding and Shen's (2019) comments regarding the "heterogeneous" (p. 3) nature of LMOOCs, in terms of topics, learning materials, and activities; but on the other, it extends such heterogeneity to degrees of CoI. In sum, the authors were able to develop a transparent, contextualized, and reliable observational protocol appropriate for measuring teaching and learning in LMOOCs.
In addition, the results of the g-study revealed somewhat different observation patterns among the rated three presences, which for a teacher's perspective could aid improvements to course design, and from an evaluator's perspective, improvements to course observation. More specifically, the variance decomposition for cognitive presence showed a somewhat different pattern from those for teaching presence and social presence. In the case of cognitive presence, course variation was lower (58.4%) than for the other two presences; whereas its rater variation and rater*course variation were higher. All of these findings suggest that reaching a consensus about the observation of cognitive presence is slightly harder than doing so with regard to observation of the other two presences. Indeed, our identification of such variance may help to explain Herrera Díaz and González Miy's (2017) finding of low scores on all four dimensions of cognitive presence. Despite class activities and learning materials being the same, MOOC learners' prior knowledge and different backgrounds (Sallam et al., 2020) may lead them to interpret levels of cognitive presence differently; and this highlights the great importance of clearly identifying more individualized learning needs (Chong et al., 2022;Hsu, 2021a;Jitpaisarnwattana et al., 2022b;Mac Lochlainn et al., 2021;Martín-Monje et al., 2018;Nie & Hu, 2018).

Identified exemplary Practices in LMooCs and Implications for Teaching
In the dimension of teaching presence, the authors identified various exemplary teaching practices in relation to design and organization, facilitation, and direct instruction of the course, confirming the importance of the design and implementation issues to LMOOC teaching success (Appel & Pujolà, 2021). As an extension to Luo and Ye s' (2021) finding that the most decisive quality factor across all types of LMOOCs is the effectiveness of teaching content, this study listed some ready-to-use teaching practices for LMOOC educators. For example, in terms of providing clear and precise instruction, it is found that it became effective when difficult content was also provided in one's local languages, similar to what Uchidiuno et al. (2018) proposed. In another example, using roundtable discussion sessions was found to be a good way to identify areas of agreement and disagreement, especially when the topic involves multiple persons with different backgrounds (Shen, 2021) and requires embracement of broad differences in LMOOCs (Mac Lochlainn et al., 2020). This study also identified concrete teaching practices to show how to provide more flexible options in a large-scale LMOOC that take account of learners' linguistic, cultural, psychological and cognitive difference (Read & Barcena, 2021), thus making a direct response to a recent call in LMOOC research regarding the individualization of teaching contents (e.g., Chong et al., 2022;Hsu, 2021a;Jitpaisarnwattana et al., 2021b;Jitpaisarnwattana et al., 2022b;Mac Lochlainn et al., 2021;Martín-Monje et al., 2018;Nie & Hu, 2018). However, at the same time, the authors found little use of innovative activities in teaching presence, such as the use of online games, vocabulary contests, and/or live chat rooms, as proposed by previous LMOOC studies (Friðriksdóttir, 2021a;Hsu, 2021b;Yaşar, 2020;Zhao, 2015).
In the dimension of social presence, although the authors have provided some exemplary practices, it is noteworthy that raters still expressed relatively moderate to low levels of social presence in the observed courses, which is consistent with previous findings that interaction was perceived as hard to achieve through LMOOCs (Barcena et al., 2015;Chong et al., 2022;Jitpaisarnwattana et al., 2021a;Jitpaisarnwattana et al., 2022c;Wright & Furneaux, 2021). This points to a need to create more innovative, or even intelligent communicative tools to aid class interaction and increase the sense of familiarity among learners (Lebedeva, 2021). For example, Uchidiuno et al. (2018) proposed a different matching strategy among LMOOC learners that is not based on time or order, but on abilities recognized by artificial intelligent systems. They also proposed to develop more intelligent systems that can moderate and personalize discussion activities. Another approach would be to encourage a frequent use of personal communication tools to interact with peers outside the LMOOC (Jitpaisarnwattana et al., 2021a), or even allow the link of one's social media account with their MOOC account (Uchidiuno et al., 2018).
In the dimension of cognitive presence, by identifying exemplary practices, the authors were able to explain what the instructor did, what the course provided, or how the course could be improved to foster students' cognitive presence of the course. Notably, we found more exemplary practices existing at the first three stages of cognitive presence, while resolution seems to show limited exemplary practices. It is consistent with the finding of Sadaf et al. (2021) from a systematic review of cognitive presence in online learning that resolution is relatively hard to achieve. This shows that more needs to be done in terms of encouraging students to walk out of their comfort zones and relate the ideas learned in the course to real-world situations (Garrison & Arbaugh, 2007;Jitpaisarnwattana et al., 2022c). Mat Daud et al. (2018) has proposed the adoption of problem-based learning and task-based learning in LMOOCs to encourage students to apply knowledge learned in the course to solve realworld problems. In our identified teaching practices, such endeavor was rarely spotted. Indeed, much of the teaching content in the sampled courses was related to academic writing, which can and should be transferred into learners' real-world academic writing practices. The authors thus advocate for more explicit instruction and design of activity on the application part of the LMOOC content that focuses on using the language in real life scenarios (Krahnke, 1987).

LIMITATIoNS
This study has several limitations. First, the selected material for analysis in this study may not include some innovative learning activities. Second, LMOOC learners were constituted of different types (e.g., viewers or all-rounders) (Martín-Monje et al., 2018) and came to class with different learning motivations (Friðriksdóttir, 2021b). It is not clear how such variation may lead to different perceptions of an effective LMOOC course simply based on the findings of this study. Third, the authors selected six courses and four units within each of the six courses to observe. This study's sample size was relatively small, and limited to courses taught in English with a focus on developing academic English skills. As such, it should not be seen as a proxy for all LMOOCs, and the authors call for more replication studies using the developed observation protocol in different LMOOC contexts.

CoNCLUSIoN
LMOOCs provide opportunities for learners to develop language-related skills via the establishment of CoIs. In this study, the authors first developed a CoI observation protocol, COILMC, to observe the existing teaching, social, and cognitive presences in LMOOCs, and tested its reliability using g-theory analysis. The results showed that COILMC is reliable, as evidenced by the large proportion of variance attributed to variation across courses rather than across raters. A follow-up d-study suggested that five and 11 raters were enough to reach moderate and substantial reliability coefficients, respectively. Based on COILMC, we were able to open the "black box" of prior correlational approaches and identified exemplary teaching practices to show exactly how the CoI framework explains what happens in LMOOCs and how educators promoted active interaction, collaboration, and knowledge construction (Palloff & Pratt, 2011). The authors believe that this will help LMOOC educators to better understand the nature of online learning and how to create and maintain CoIs in that context. Additionally, the findings highlighted the value of using observational studies to disentangle the dynamic interchanges that occur in LMOOCs, as a means of providing insights into the findings of traditional correlational studies. The results also show how the different CoI presences may be best observed using somewhat different techniques, which may inform improvements in instructional design and teaching practices. Last but not least, the authors hope that the instructors and designers of LMOOCs will use COILMC to observe, design, and teach LMOOCs in the future.

CoNFLICT oF INTeReST
The authors of this publication declare there is no conflict of interest.

FUNdING AGeNCy
This research was supported by the Ministry of Education in China Humanities and Social Science Project [grant number 20YJC880126].

APPeNdIX 1
The development of the observation protocol 18. All learning activities enabled students to humanize the instructor as real. Examples: a. Incorporates audio and video within the course content b. Shares personal stories, professional experiences c. Addresses students by name d. Shows teachers' character and personality e. Exhibits a sense of humour f. Uses emoticons 19. All learning activities enabled students to humanize peers as real. Examples: a. Creates a 'meet your classmates' section b. Encourages students to share experience and beliefs c. Uses emoticons open communication 20. The instructor provided adequate chances for students to converse through the online medium. 21. A feeling of comfort could always be drawn throughout the course discussions. 22. The instructor always encouraged students to express themselves.

Group cohesion
23. Even showing disagreement, the course still maintains a strong sense of trust. 24. Students' point of view was always acknowledged by other course participants. 25. Collaboration among participations was always encouraged throughout the course.

Triggering event
26. This course always posed effective questions or problems to increase learning interest. 27. Course content or activities piqued learning curiosity. 28. The course consistently took students' previous learning into account when introducing new knowledge. exploration 29. The course provided adequate diverse information sources/activities for students to explore authentic settings. Integration 32. The course effectively integrated new information to help answer questions raised in course activities. 33. Learning activities are helpful for students to construct explanations/solutions. 34. The course provided adequate opportunities for reflection on course content and discussions.

35.
The course always encouraged students to apply the knowledge gained in this course to authentic settings 36. The course developed effective solutions to course problems that can be applied in practice.