Measurement of Student Engagement in a Generic and Online Learning Management System-Based Environment

Use of technology on campuses of higher education has changed how students are engaged in the process of learning. It also has brought lot of asynchrony to the definition of class room teaching, active learning, student-staff interactions, and dealing with various academic challenges. The paper presents a study conducted to measure student engagement in a generic and online learning management system-based teaching learning environment. The paper presents threefold results. The first one is on identification of student engagement styles. The styles identified can further be used to design, develop, and implement most student engaging policies on campus which are beneficial to all the stakeholders. Second, the central point of the study is the student and her/his engagement with the learning process. The paper presents a student engagement report card to individual students for their analysis. Informing and involving students to know about their engagement report card would be beneficial. The third is feedback on a trail left by students’ logs on the learning management system that can help the teachers to plan the teaching methodology. The methodology used was based on the data collected by the students of the institute/university. A student engagement questionnaire was used to measure student engagement in both generic and online learning environments. A cluster analysis was conducted on the data collected to identify the student engagement styles. A subcategory analysis was reported as a student engagement report card. The student-logged data on the institute learning management system was used to present the third analysis.


INTRODUCTION
Student engagement is used to, "depict students' willingness to participate in routine academic activities, such as attending class, submitting required work, and following teachers' directions in class" (Kuh (2001)).It has even been suggested that student engagement could be used as an indicator of institutional teaching quality (Kuh (2001)).In higher education, engagement has become a catch-all term most commonly used to describe a compendium of behaviours characterizing students (Krause (2005)).
The time students spend on learning tasks and willingness to participate in these activities can be considered as engagement.(Stovall (2003)).Engagement is quality of effort students themselves devote to educationally purposeful activities that contribute directly to desired outcomes.Krause and (Coates (2008)).Additionally, (Chen, Gonyea and Kuh (2008)) say that engagement is the degree to which learners are engaged with their educational activities and that engagement is positively linked to a host of desired outcomes, including high grades, student satisfaction, and perseverance.
However, the term is also increasingly used to describe meaningful student involvement throughout the learning environment, including students participating in curriculum design, classroom management and school building climate.(Bomia et al. (1997)).It is also often used to refer as much to student involvement in extra-curricular activities in the campus life of a school/college/university which are thought to have educational benefits as it is to student focus on their curricular studies.(Chapman (2003)).
The current research attempts to measure student engagement in generic and online environment.Such a measurement leads to group students and provide customised learning solutions as per their student engagement styles.A typical report card can be prepared so as to help all stakeholder for supporting the development and providing access to learning environment at Higher Educational Insitututions (HEI)s.The current paper is organised as, in section 2 details on the student engagement, it's measurement and learning environment and need for such research.Section 3 presents methodology used to present the research results and section 4 and 5 presents the results, discussions and conclusions.

RELATED wORK
Student engagement research works from the perspective that education is fundamentally about students creating their own knowledge.While students are seen to be responsible for constructing their knowledge, learning is also seen to depend on institutions and staff generating conditions that stimulate and encourage student involvement (Davis and Murrel, (1993)) For the last two decades or so, the penetration of sophisticated technology tools like online learning management systems into many educational institutions have provided a blended learning environment for the current student generation.Learning management systems (LMS) are at the forefront of this technological development.LMS have been designed to have a diverse influence on the ways in which campusbased students engage with their university studies (Hamish Coates (2006)).LMS have the capacity to change how students collaborate with others, communicate with staff, and access the materials which they use to learn.It enriches student's learning experiences and engages them by opening wide range of resources.However, on this important and emerging aspect of higher education very little work has been reported.Research on this aspect comes from National Survey on Student Engagement (NSSE), USA, and also from Australian Student engagement research studies.In India, the National Mission of Education Information Technology Communication (NME ICT), an Indian government body, is also conducting various studies on student engagement.
Before investing on the online infrastructure at Higher Educational Institutions (HEIs), there is a need to understand what themes are important for online student engagement experiences.In a study conducted by authors Farrell and Brenton, five central themes such as student's confidence, learning approach, sense of community, supporting network and balancing life activities and study were proposed.They found that students developed time management and organizational skills in an online setup.They also have reported that successful online engagement enabled a perfect balance of life load and study for students.(Farrell and Brunton, (2020)).
In another interesting study authors showed significant results for student engagement as a mediating variable for perceived measuring of student learning, course content and structure, learner interaction, instructor presence.The paper also presented a relationship among the aforesaid four parameters.The study was intended to show the quality of online teaching and learning.The model was tested on Amos 23 with the data collected by graduate students.(Gray DiLoreto (2016)) Moore's interaction model (Moore and Anderson, (2003)) was tested by the authors (Martin and Bolliger, (2018)) for different student engagement strategies used in online courses.Some of the strategies considered were between Learner (L), Content('C), Instructor (I).The participants in the survey valued the L to I derangement strategy.
Research in the area of LMS based learning is in evolving stage.A large number of studies available in this area focuses on technology transformations, pedagogy, faculty work and academic development.Learning and student experiences have been addressed in highly compartmentalized and particular issues.Most of these studies focus on a particular aspect like email use (Gatz and Hirt, (2000)), website navigation (Selim (2003)), and learning styles (Stokes (2001)).
While the degree to which LMS are affecting student engagement in universities is not clear, the importance of engagement is established in the literature and therefore further research into measuring engagement within LMS is warranted in order to identify and address inhibitors that LMS place on engagement.The present paper therefore combined measuring of online and generic student engagement in a HEI where both the facilities were provided.The details of the work are presented in the next section.

PROBLEM FORMULATION AND METHODOLOGy
While technology has entered the campuses, there is a need to understand how student would behave differently for general or traditional learning Vs online learning.Therefore, a blended learning set up in the selected HEIs was essential for the study.In the present paper, the authors focus on the measurement of Student Engagement in such a setup with a generic setup and online learning environment.

Student Engagement Measurement Constructs
Objective is to test the models traditional student engagement and online student engagement on students as subjects that meet the criteria as defined by the researchers.While there has been lot of research that is available on student engagement over the past few years, the most well defined framework has been developed by USA's National Survey of Student Engagement (NSSE (2000(NSSE ( , 2012)); Kuh (2003)).The framework divides student engagement into six benchmarks each under the general and online learning environment.
Figure 1 shows the six constructs considered under student engagement -General.This framework has been evolved out of substantial amount of research into good or effective practices in As the blended learning strategies including LMS and generic, the online environment provided the higher educational institutes also needs to be studied from online student engagement point of view.The constructs contributing this are shown in Figure 2 and these are Online Engagement, Online Active Learning, Online Academic Relevance, Online Teaching, Online collaboration and Online Social Interaction.

Research Objectives and Problem Formulation
The objectives of the study were based on seeking answers to the following research questions: RQ1: How to measure student engagement under generic and online learning management system based environment?RQ2: What online activities benefit on an E LMS help improve student engagement?RQ3: How can a student benefit from student engagement measurement?
To answer the Research Questions (RQ1 and RQ2) an instrument based on student engagement, Student Engagement Questionnaire (SEQ) was used and administered.A standard questionnaire based on well-defined framework from USA National Survey of Student Engagement has been customized for capturing the data for the study.(NSSE, (2000(NSSE, ( , 2012) ) Kuh (2003)).The instrument has evolved out of substantial amount of research into good or effective practices in university education.
The Student engagement measurement was operationalized using Hamish Coates (2006) and Collin Beers (2010) and NSSE 2012) theoretical framework and referred in this chapter as Student Engagement Questionnaire (SEQ).The questionnaire has standard constructs tested by researchers both for their validity and reliability.The focus was on multidimensional measurement of online and general student engagement with 103 items speared over online and general engagement.Online engagement with 50 items measured the six key qualities.Online Engagement (OE), Online Active Learning (OAR), Online Teaching (OT), Online Collaboration (OC), Online Engagement (OE) and Online Staff Interaction(OSI).Similarly the another six key qualities of general engagement with 53 Items are also measured.Academic

Sample Description and Data Collection
The sample for the study was students studying at different institutes affiliated to the same university.
The unit of measurement for the study was individual users of blended and/or e -LMS based environment for their respective course/learning engagement.Our institute being one of the early adopters of blended learning environment, the natural choice for the sample selection was students of our institute.We also expanded our student sample base to other sister institutes so that a cohort and cluster analysis could be carried out to understand both online and general student engagement on different university campuses.The sample is described in Table 1 describes the same.
The data was collected majorly by using online methods.The SEQ was available for the duration on our e-LMS server spread over two semesters during June 2014 till April 2015.
The data from our students was collected on our LMS platform during the same duration.For students from other sister concerns, Google Drive and Google forms were used.About 10% of the data was also collected manually.

RESULTS AND DISCUSSION
A cluster analysis was carried out on the data collected using K means algorithm.This was useful in profiling the students based of their engagement coefficient.Also a sample of individual student engagement report card has been generated using the data.The study also presents analysis of digital trails left by the student on the institutional LMS.The results are arranged in three sections.Section 4.1 discusses the results of cluster analysis and identification of possible student engagement styles.Section 4.2 describes a Student engagement Report Card.Section 4.3 analysis and discussions on student logs collected on institute LMS.

Student Engagement Styles
The sample data of about 509 was considered for identifying possible similarities among the units of measurement.So, we wanted to group them or classify them in different categories.There are number of methods that can be used for classification.(Clustan (2001), Coates (2007)).The first step involves classifying students into groups.The present group used K-means algorithm.Cluster analysis was the main statistical tool used to investigate the relationship among student engagement.The results were used to identify different patterns and hence classify students in terms of different engagement styles.
The purpose of cluster analysis was to categorize N(509) objects in k (k>1) groups, called clusters, by using p (p>0) variables.The number of variable p for each construct was different.The K-means algorithms iterative process led us to classify cases into groups that are relatively homogeneous within themselves and heterogeneous between each other, on the basis of a defined set of variables.These groups are called clusters.The method involves following steps.
The pseudo code for the algorithm is as follows: Start { 1. Initialization of number of cluster and convergence conduction.
2. An initial set of k "seeds" (aggregation centers) is provided First k elements Other seeds 3. Given a certain threshold, all units are assigned to the nearest cluster seed 4. New seeds are computed 5. Go back to step 3 until no reclassification is necessary end if no more classification is required.}End A number of iterations with different algorithm parameter settings were considered to test both online and general SEQ scales.To extract information about the pattern two to five clusters were considered.Two being the minimum number of clusters five is close to maximum SEQ scale value.
The pattern that emerged after the iterative process showed clear indication of definable four clusters.Two and three clusters had an overlapping cluster mean values compared to four clusters.Also, the patterns seemed to provide insufficient information for distinction.Cluster five had produced redundant patterns and uneven sized values for clustering.Moreover, using four clusters a distinction among the groups and interpretation was possible.Therefore, for further analysis four clusters were considered.The combined clustering could converge in about 15 iterations and general and online engagement algorithms separately converged in 9 and 14 iterations respectively.
Scale means for general and online SEQ are shown in Figure 3(a), and Figure 3(b).Mean value for each construct for every group is deliberately joined to form a curve.Each curve therefore represents one cluster.Every construct point on the respective curve is drawn by taking into the difference between cluster mean value for the selected construct and full sample mean value for that construct.
The groups exhibited some interesting characteristics.One of the clusters remained above the group mean value and another group exhibited way below the average value.
In the explorative study conducted the levels of engagement converged above average levels were stated as (Coates (2008)) Intense and students who were below the average levels are labeled as Passive engagement styles.The two middle groups presented an interesting style replicating across the analysis.They described two broad aspects of social and academic concepts of the university study.One group indicated slightly above average response on academic concepts like (AC, ACL, OAL, OC) but the same group have shown slightly below average response on interaction or supportive learning environment .This behavior may be attributed to their passive goal settings but active in academics.This cluster is therefore referred to as Academic.The other middle group has shown a converse picture of the earlier group.So, they are referred as social style.This group in general style exhibited more focus towards employability and lower focus in all online constructs except online engagement.Online engagement measured number of times student used online system for learning and evaluation.So, the higher value of online engagement for this group could be attributed to institutes LMS policies for learning and evaluation.
A cross-tabulation between the general and online analysis indicates an independent style of student engagement towards their university study.While the students tend to be classified into similar groups but considerably many students change their groups between online and general clusters.It also therefore exhibits the independence of both online and general engagement styles.The average similarity in online to general engagement is about 6% whereas for general to online engagement is about 10%.The cross tabulation results are shown in table 3 for four cluster analyses for the complete sample.

Student Engagement Report Card
A study in continuation was performed on a homogeneous sub sample.The objective was to report individual level of engagement.This may be useful under multiple settings.Figure 2 shows one such report card.The report card display both online and general engagement scores.
The report card is prepared as follows: 1. First step is to identify the student engagement style.For this, the SEQ records of a homogeneous group-Test Class is chosen with a sample size as 50.A cluster algorithm as discussed earlier was applied to group them into groups.The clustering was done separately for online and general dimensions.This enabled us to identify online and general engagement styles of each participant in the homogeneous group.A cross tab for the same in shown in Table 3. 2. The second step is to prepare a report card.To display the report card a comparative analysis with the class and his/her own cluster was considered.From the student records, the mean value for each construct for every student could be displayed along with cluster and class mean values as shown in   c.Student would like to discuss his/her course and career goals with concerned staff (generic 3.25/5 ; online 4.8/5) but prefers to do so more frequently on online rather in person.d.The Intense style for online learning is justifiable with scores although he still struggles to find out the academic relevance of online systems.In all other online constructs his participation is more than 75%.e.A deeper analysis into the SEQ records made us understand why the score for academic relevance engagement was low?

Measurement of Student Logs on Institutional LMS System
The LMS logs for two cohorts were captured and analyzed for any possible inference.The entries in Table 4 has total number of visits by the respective students of the class to each course page.The visit could be just view quiz marks, appear for quiz, see attendance or discuss with peers and teachers.Out of all these courses XML technologies, Server Side Web Technology (SSWT) and Computer Organization (CO) were taught by the faculty members who were part of this research team.The team used different blended learning techniques like course forum creation, glossary creation, WIKI and lesson creation on the institutes LMS.The enrolled students were given different rights of view, add and update.Every student has a right to appear for an online quiz and check his current attendance status for each courses.Table 5 indicates the division of these hits for UG top three LMS hits based courses for comparison.Course taught using blended learning strategies (SSWT and CO) made students spend more time on LMS, in adding content either as wiki or discussion in the forum or creating a glossary etc online.This also gave enough opportunity for students to learn from anywhere and anytime.

CONCLUSION AND FUTURE RESEARCH DIRECTIONS
In summary, the results of the investigation to search answers for RQ1 to RQ3c, supported the finding that it is possible to measure student engagement for online and general (face-to-face) education.Data collected showed distinct engagement styles referred here as Intense, Academic, Social and Passive.Interestingly, as assumed student exhibited different engagement style for online and general education.A student engagement report card could be generated, discussed for the benefit of all the stakeholders of higher education.It was found that the courses which were taught using different online strategies have attracted more student time online in doing various course related activities.
The investigation was carried out in a particular set up.The scores received were dependent on the adapted benchmarks.These may not be the enduring qualities of student engagement that are sustainable over time and across contexts.A longitudinal investigation could help researchers understand how processes change in individuals and detect developments or changes in the characteristics of the target population that might occur over a period of time.

Figure 3 .
Figure 3. Student engagement groups: (a) General, (b) online Figure 4. 3. The selected student had exhibited "Academic" learning style for general whereas "Intense" under online style.Some of the interesting analysis of these observations are: a. Selected student clearly has shown an above average response in both online and general category.b.By the score values on enriching educational experiences (score value 4.25/5), it could be understood that the student frequently collaborates with others in the class with diverse backgrounds, race and religion, with others outside the class and with others on online platforms.(Score value 4.4/5).

Figure 4 .
Figure 4. Student engagement report card

Table 5 . Student actions for select courses
Rajashree Jain is a Professor at Symbiosis Institute of ComputerStudies and Research, Pune, India.She has more than 25 years of experiences in teaching, research and academic administration.She obtained her Ph.D Degree under the faculty of Computer Studies from Symbiosis International University, Pune, Maharashtra, India.She holds double Masters Degrees(M.Phil and M.Sc)in Applied Electronics from Jnana Ganga, Gulbarga University, Gulbarga, Karnataka, India.She has been a meritorious student all along and was a national merit scholarship holder, a rank holder and has received a Gold Medal for securing first rank during her post-graduation.Her research areas are Microwave techniques, antennas, optimization, nature based algorithms, AI and Sensor network, IOT, Data analysis, entrepreneurship and education.She also has more several research papers to her credit published in reputed Journals and has presented research papers at various national and International Conferences.She has served as Chair/member of organizing various technical conferences and leadership summits.She is an active member of professional organizations like IEEE, CSI and National Entrepreneurship Network (NEN).Past several years she is been working as an active volunteer to serve IEEE Pune Section in various capacities.She currently holds Vice chair, Membership Development, IEEE India council.She was Vice Chair, Women In Engineering (WiE), IEEE India Council, 2016-18, Chair, Women In Engineering, IEEE Pune Section 2014-2016.Under her leadership, WIE AG Pune received WIE AG of the Year, Honourable mention award for the year 2016 from global WIE.Minal Kashinath Abhyankar has Master's Degree in Computer Science from Savitirbai Phule Pune University Pune.She is currently a senior software developer working in the Quality Measurement and Benchmarking department of Symbiosis International Deemed University (SIU), Pune.She has more than 15 years' of experience in web development technologies and implementing MOODLE a learning management (System) at SIU.Her areas of interest are Software usability, Learning Management System and end user experience and satisfaction, teaching learning process, Database and data mining and Fuzzy logic and artificial Inteliiengence.Chair Signal Processing, IEEE Bombay Section, and an active member of Machine Intelligence Research Labs and USERN (Universal Scientific Education and Research Network) (April 2020 -present).He is one of the active reviewers for Wireless Networks, Journal of Springer Nature.His name is listed at 19th position in Top 100 Private University's Authors Research Productivity Rankings given by the Confederation of Indian Industry (CII) based on the "Indian Citation Index" Database 2016.Rajanikanth Aluvalu (Senior Member, IEEE) received the Ph.D. degree in cloud computing as specialization.He is currently working as Professor, Department of IT, Chaitanya Bharathi Institute of Technology, Hyderabad, India.Served IEEE Hyderabad section as Vice-Chair of the Entrepreneurship and Startup Committee, Treasurer and the Secretary of the IEEE Computer Society.He is having more than 19 years of teaching experience.He organized various international conferences and delivered keynote addresses.He has published more than 100 research articles in various peer-reviewed journals and conferences.He is a Life Member of ISTE and a member of ACM and MIR Labs.He is certified international engineering educator by IUCEE.He was a recipient of the Best Advisor Award from the IEEE Hyderabad Section as well as the IUCEE Faculty Fellow Award (2018).Editorial board member of IJDMMM journal published by Inderscience.Mahesh S. Raisinghani, is a Professor of Management Information Systems in TWU's College of Business; a Senior Fellow of the Higher Education Academy in the U.K., and a Director of Strategic Partnerships for the Association of Information Systems SIG-LEAD.Dr. Raisinghani was awarded the Distinguished Research Award by the Association of Business Information Systems in 2022, ISACA's 2021 Global Excellence in Education award, TWU's 2017 Innovation in Academia award, UMGC's Excellence in Teaching award in 2017, the 2015 Distinction in Distance Education award, the 2008 Excellence in Research & Scholarship award.He has edited eight books, published over 250 manuscripts in peer-reviewed journals, conferences, and book series.Dr. Raisinghani serves as the Editor in Chief of the International Journal of Web-based Learning and Teaching Technologies; on the board of the Global IT Management Association; advisor for the National Society of Leadership and Success chapter at TWU; and as an advisory board member of Enactus and X-Culture.org.He is included in the millennium edition of Who's Who in the World, Who's Who among Professionals, Who's Who among America's Teachers, and Who's Who in Information Technology.
MVV Prasad Kantipudi is working as anAssociate Professor in the Dept. of E&TC, Symbiosis Institute of Technology, Pune.He received his B.Tech.(Electronics and Communications) (2009) & M.Tech.(Digital Electronics and Communication Systems) (2011) degrees from Jawaharlal Nehru Technological University, Kakinada.He received his Ph.D. (Signal Processing specialization) from BITS, VTU, Belagavi (2018).He, previously, worked as the Director of Advancements for Sreyas Institute of Engineering & Technology, Hyderabad, and also as an Associate Professor with R.K. University, Rajkot.He is having teaching experience of around 11.2 years.His current research interests are in Signal Processing with Machine Learning, Education and Research.He is recognized as a technical resource person for Telangana state by the IIT Bombay Spoken tutorial team.He conducted key Training Workshops on Open-Source Tools for education, Signal Processing and Machine Learning focused topics, Educational Technology, etc.He has authored and co-authored many papers in International Journals, International Conferences, National Conferences and published five Indian Patents.Prasad is a Senior Member of IEEE (Membership ID: #93513961) and serving as a