A Study Exploring Different Modalities to Integrate Learning Objectives in Games

This research aims to provide further insight on how to design effective educational games by exploring whether the integration of educational content through game mechanics, text, or a combination of both text and game mechanics is more effective in teaching the learning outcomes in games. The results of the study show that all three methods led to information assimilation. The study showed that the participants did not necessarily learn better through a combination of text and game mechanics as compared with those who were exposed to learning objectives integrated into the game only through text or game mechanics. Some learning objectives were better learned when they were integrated through text while others through game mechanics.


INTRODUCTION
The design of educational games is an inherently difficult problem.Combining two apparently contradictory aims fun and learning makes it a complex problem which requires further exploration.Research literature report mixed results regarding the effectiveness of the educational games (Checa & Bustillo, 2020;Molnar, 2019;Silva et al., 2021;Tokac et al., 2019) and how to design them in such a way they are effective is an area that requires further research.Literature reports on the lack of empirical evidence on what and how elements of the game affect educational content assimilation (Boyle et al., 2016;Fanfarelli, 2020).Games are seen as separate educational methods that do not neatly fit with the existing educational theories and frameworks (Smaldino et al., 2011).Moreover, it has been found that pedagogical expertise is not necessarily easily translated into game design (Theodosiou & Karasavvidis, 2015).Therefore, a better understanding on what works better when designing serious games could help as game designers and developers are left with "almost no guidance […] on how to design games that facilitate learning" (O'Neil & Perez, 2008).The lack of methodologies on how to design games is also reported in Callaghan et al. (2016).This paper aims to address this gap, by examining what is more way of delivery and integrating educational content in serious games: text, game mechanics or both.
In this paper we define game mechanics as "the procedural mechanism of a game that provides the essential interactions required to create a meaningful game activity" (Habgood & Ainsworth, 2011).Game mechanics include elements (e.g. a score, movement) and rules (i.e.how the game is played) (Arnab et al., 2015).
The educational content is intrinsically integrated (Kafai, 1996) in the game regardless of the method through which it is delivered: text or game mechanics.The text is seamlessly integrated in the game narratives and the game mechanics through which the educational content is delivered and is connected with the other mechanics of the game and with the narratives.Game mechanics are essentially present in every game genre, whereas narratives can be a predominant part in some genres such as storytelling-based games (Molnar & Kostkova, 2016), but they can be completely missing in others.However, when it is part of the game design, text can convey educational messages through traditional game design opportunities such as mission briefings, in-game conversations (Molnar, 2018).
Teaching through game mechanics requires that the player interprets game dynamics correctly and discovers and constructs the educational message with less clear directions.The appeal of this approach lies in understanding constructivist learning theories (Hein, 1999), where individuals learn by constructing knowledge in this case through the interactions with the game.In this case the learner does not become a simple recipient of knowledge, but he becomes in charge of its own learning.
On one side, game mechanics by being embedded in the game could lead to a fear that the students may fail to construct knowledge from a "hidden curriculum" (Starr, 1994;Turkle, 2003).The transfer of the knowledge accumulates through the game, between the game context and a different context (i.e.classroom) is not always present (Bavelier et al., 2011).On the other side, game mechanics could provide a better learning experience than text and better engage the player with the content.They could allow the player to construct the knowledge which is known to be a better way to promote "deep learning" and long-term retention (Sawyer, 2015).However, a combination of both text and game mechanics could help with the "hidden curriculum" and disambiguate, but also promoting "deep learning" through the game mechanics.
Previous work has investigated the differences in how the students learn with text and game mechanics and have shown that the relationship between text and game mechanics complex and ambiguous (Molnar & Kostkova, 2013).We want to build upon this work and see if a combination of both text and game mechanics could lead to better outcomes than only text or only game mechanics.This would help to get closer towards an understanding of how to design serious games.
The rest of this paper is organized as follows.The next section presents previous work in educational game design.Afterwards, the study design is presented followed by the evaluation results.The last section ends the paper with our conclusions and future work.

BACKGROUND
How to design educational games in a way that they are both entertaining and effective in teaching is still an area which requires exploration (Bavelier et al., 2011;Mayer, 2015;Wilson et al., 2009).Various studies have attempted to find out how to design educational games (Callaghan et al. 2016;Fanfarelli, 2020;Kelle et al. 2013;Lepe-Salazar, 2015;Molnar & Kostkova, 2013, 2015;Novak et al. 2016;Novak 2015), focusing on different aspects of the game design.
Several researchers have focused on looking into what design patterns have been effective in educational games (Lepe-Salazar, 2015;Kelle et al. 2013).Lepe-Salazar (2015) investigate design patterns that have been used in successful educational games and conclude that future games should consider the goal of the game, the audience, logical and technical aspects of the game and the environment where the users will play.Kelle et al. (2013) evaluated two different design patterns: timer pattern and a score pattern on a healthcare game.They show that the combination of both patterns tends to show positive effects on knowledge gain and user experience, especially for the older participants.
Other researchers have used on the usage of game mechanics and storytelling within the game.Fanfarelli (2020) researches the impact narrative and badging have the players' engagement with an educational game.The results showed that the narratives and badges did not have an impact on either learning or engagement.Molnar & Kostkova (2015) looked at how to integrate learning objectives within interactive digital storytelling-based games and proposed a framework (STAR) focused on four components four main components: introduction, puzzles, resolution, and debriefing.The framework has been integrating within several interactive digital storytelling-based games.Novak et al. (2016) shows that the introduction of storytelling within a simulations did not have any effect on learning as compared with the same simulation without the storytelling element.Callaghan et al. (2016) advocates the usage of game mechanics and mapping of each of the learning mechanics to a game mechanics in the game, whereas the narratives are used just to set the game within a context.A similar approach is advocated in Arnab et al. (2015), whose study aimed to map game mechanics to pedagogical principles Molnar & Kostkova (2013) assess which method is the most effective in integrating the educational content in game: through game mechanics or through text.They found out that the relationship is not straightforward and more complex.Similarly, a critical review of the usage of the storytelling in games for learning purposes shows mixed results regarding its effectiveness compared with the non-storytelling environment (Novak, 2015).
Therefore, there is a need to further understand what game characteristics makes them more effective for educational purposes.In this study, we focus how a combination of text and game mechanics used in delivering the learning outcomes compares with learning outcome delivery through text only or through game mechanics only.As opposed to Molnar & Kostkova (2013) we do not compare text and game mechanics only but we look further into the combination of text and game mechanics and how this compares with only text or only game mechanics.Sawyer (2015) state that for the learning to be effective, educational games must achieve a state of "deep learning", which can be achieved through certain principles of a learning theories such as constructivism.Learning through game mechanics fits into the principles of the constructivism, as the player learn by interacting through the game mechanics and construct knowledge from the interactions.On the other side, narratives or text integrated through the game promotes a passive learning by delivering the information to the learner.H1: Learning through game mechanics is more effective than learning through text.

HyPOTHESIS DEVELOPMENT
Integrating the same learning objectives both through the text and game mechanics would reinforce the knowledge.As repetition helps enhance learning, this should improve the learning outcomes are compared with the integration with only through game mechanics or only through text.H2: Learning through both text and game mechanics is more effective than learning through game mechanics only.H3: Learning through both text and game mechanics is more effective than learning through text only.

GAME
Bugs Kingdom (Molnar & Kostkova, 2013) is one of the educational games from edugames4all.org platform.The game aims to teach children about microorganisms' transmission, hand and food hygiene and responsible antibiotic use.The game is based on the educational curriculum for primary schools' children, and the learning objectives covered in the game have been selected through the help of the experts in the field.The game design followed a user centred design (Abras et al., 2004) approach, with students being involved from the initial stages of the game implementation (Farrell et al., 2011).The game elements are design in such a way that they represent the reality as closed as possible, but they are also appropriate and pleasant for the age group.For example, all microorganisms in the games are design based on the real representation of the microorganism.
In most of the game levels, the game is shrunk to the size of a microorganism.In this way s/he is observing and interacting with the different microorganism s/he encounters.The player travels on different locations such as human hand or inside the human body.The player is teleported in different locations using a portal if s/he manages to surpass the challenges in the game.
Learning objectives are taught through the game based on a combination of bespoke game mechanics and text.For example, teaching that the certain microbes are used in the process of making yogurt, the student must push lactobacillus bacteria into a glass of milk and the milk transforms in yogurt (see Figure 1 and Figure 2).The learning objectives are reinforced through either executing the same game mechanic in different contexts or through text (see Figure 3 for an exemplification on how the text is used in the game).

In-Game Assessment
The game has the advantage of having the assessment seamlessly integrated in the game (Kostkova & Molnar, 2014).The in-game assessment is done through a quiz provided to the player before and after each game level, in such way that it assesses the player knowledge of the integrated learning objectives before and after playing the level.The game quizzes are designed similar with "How to be a Millionaire" television show.The quiz consists of a facilitator who asks questions and another virtual character who plays against the player.The game questions allow the player three options: "agree", "disagree" and "don't know".
Although both the evaluation performed before and after the playing a level can provide the player for feedback, to not affect the learning outcome, the quiz provided before the game level does not provide feedback to the player when they provide a wrong answer.However, the final quiz provides feedback in this way also reinforces the learning objectives provided through the game.For evaluation purposes only we learned through the actual game and not through the evaluation game, no feedback is provided in the first round, called "Blind Question Round", because this round is provided before the actual game play took place.However, feedback in the form of whether the questions are right or wrong is provided during the second round, which is performed at the end of the game level.
This assessment method has been previously evaluated (Kostkova & Molnar, 2014) and the results have shown that it is preferred to fill a questionnaire by most of the players.It has also shown that, as measured by the enjoyment questionnaire, the difference in the players' answer, between those playing the game without the quiz and the ones with the quiz, was not statistically significant.

Game Levels
The game has five different levels which teach deliver different educational outcomes and healthcare messages (Farrell et al., 2011;Molnar & Kostkova, 2013): • Introduction to Microbes: teaches the player about the different types of microbes • Harmful Microbes: the players meet "harmful" microbes for humans and they have to fight against them • Useful Microbes: teaches about the importance of "good" microbes and how they are used in day to day life • Food hygiene: the players learns about the importance of the food hygiene and how the food needs to be stored • Antibiotics: focuses on responsible antibiotic use.
Playing the game fully takes a long time and it is not achievable in a single session.Previous experience with the game has been shown that when trying to have a single session to play all the game, the players drop out before the end of the game (Farrell et al., 2011).Due to these reasons, it was decided that for the study, two levels of the game will be used.The two selected levels were Harmful Microbes and Useful Microbes.They are teaching similar learning objectives and through

Learning Objectives
Both levels of the game covered several learning objectives.However, for the purpose of the study we decided to select a combination of learning objectives which have shown to be either very effective in teaching players (the knowledge of the players improved as a result of playing the game) or not.The selected learning objectives from the two levels: Harmful Microbes and Useful Microbes selected were similar with Molnar & Kostkova (2013): • LO 1: Soap can be used to wash away bad bugs • LO 2: Our bodies have natural defences that protect us • LO 3: We use good microbes to make things like yogurt • LO 4: All microbes are bad for us LO1 and LO2 are part of the Harmful Microbes level and LO3 and LO4 in Useful Microbes levels.The original game levels were modified in such a way that three versions of each level have been created: • One version which teaches the learning objectives just through game mechanics • One version which teaches the learning objectives just through text • One version which teaching the learning objectives through both game mechanics and text.
The integration of the learning objectives through text was performed by having a message delivered through the player on the mobile phone similar with the example presented in Figure 3 for the LO 4.
The game mechanics used to teach each of the LOs are explained below: • LO 1: Soap can be used to wash away bad bugs: -The player is initially shrunk to the size of a microorganism and is teleported on a human hand.There he can see the microorganism and the dirt on the hand.The goal of the journey is to clean the hand.To achieve his aim, he must collect soap and throw bubble soap at the harmful microorganism that s/he encounters (see Figure 5).If the player is touched by a harmful microorganism s/he loses one of his game "lives", however if the soap touches the microorganism, it disappears.• LO 2: Our bodies have natural defences that protect us: -As in the previous game mechanics, the player is shrunk to the size of a microorganism.In this case s/he is exploring the inside of the human body.Through the journey the player encounters harmful microorganism.In order to protect the body and the player not to lose his game "life" s/he has to collect white blood cells and throw them at the harmful microorganism (see Figure 6).• LO 3: We use good microbes to make things like yogurt: -In this case the player is in the kitchen and again the size of the player is smaller than in reality.While exploring the kitchen the player encounters lactobacillus bacteria, which when pushed into a glass of milk, they transform it into yogurt (see Figure 1 and Figure 2).• LO 4: All microbes are bad for us: -From the previous game mechanics, the player has to infer that not all the microbes are "bad" microbes.

METHODOLOGy
In this study, we focused on how certain elements of the game influence the effectiveness of the game in delivery the learning objectives.The only way to understand what relationship exists between different game components is to study them in isolation (Wilson et al., 2009).Therefore, we designed three versions of the same game: 1. one delivering the objectives through narratives 2. one delivering the learning objectives through text 3. the third one delivering the learning objectives both through text and game mechanics.
This research aims to compare the results of the learning assessment obtained across these three different conditions and to see whether any of them provided better performance.
The study followed a quasi-experimental (Cook, 2015) and between subjects' design (Charness et al., 2012).A within-subject design would have not been possible here as playing the same game different times would have affected the learning achievements and it would have been difficult to determine if a different version of the game did help or not with the learning.Student t-test (Petscher et al., 2013) with a 90% confidence interval was used for determining statistical significance.
To collect data regarding the learning achievements, we use the already build in-game assessment (Kostkova & Molnar, 2014).The in-game assessment provided us with the knowledge of the learning objectives before and after the game, as well as a post assessment of the learning objective after each game.The difference in player performance (defined here as knowledge improvement before and after the game) was measured using descriptive statistics.
The subsection below discusses the participants and set-up for the two different studies carried out: • text or game mechanics • text and game mechanics.
The reason for organising two different studies was to be able to recruit more participants.

Text and Game Mechanics
Harmful Microbes and Useful Microbes game levels were played by the participants in this study.
The learning objectives presented in the section above were incorporated into the game through both text and game mechanics.A total of 154 participants played both levels.The participants were recruited from various primary schools across the UK.The students played the game either in the classroom or online.

Text or Game Mechanics
As in the previous case, Harmful Microbes and Useful Microbes game levels were played by all the participants of the study.However, there were two versions of the same game levels, one which had the learning objectives taught through game mechanics and another one through text.This experimental study followed a between-group design (Charness et al., 2012).A total of 61 participants volunteered to take part in the study.Their age ranged from 6 to 12 years old.The participants were randomly divided between the two versions of the two-game levels (i.e., 31 children play a version and 30 another version).

Results
The results are divided across two different sections: 1. Ability to learn: covering the potential of the different means to integrated learning in the game to teach 2. Effectiveness: covering whether one means was more effective in teaching than the other.

Ability to Learn
The first part of the study focused on student ability to learn with the three different game options.Student t-test was applied to the participants' responses to the questions before they are exposed to the game mechanics and the ones afterwards.The results show that the participants learned regardless of the means through which they were taught, and statistical significance has been obtained across the same learning objectives, considering a confidence interval of 10%.Table 1 presents the p-value for all three means of teaching.In all of the cases, a statistically significant difference has been found for the first learning objective: "Soap can be used to wash away bad bugs" and last learning objective: "We use good microbes to make things like yoghurt".

Effectiveness
The second part of the study assessed whether one method of delivery is better than others.In doing so we use descriptive statistics.Figure 7 presents the percentage of players improved their knowledge as a result of the game playing session relative to the number of people who did not provide the right answer in the pre-questionnaire.We consider that players did not provide a right answer if s/he responded in the prequestionnaire with a wrong answer or selected the Don't Know option.The data is provided relative to the number of players who did not know the correct answer at the beginning of the game playing session, as the samples were not necessarily comparable: more players in the narratives and game mechanics version of the game know the correct answer before playing the game than in the other two cases.
Although all the methods teach the player, combining both text and game mechanics does not necessarily perform better.The results between delivering the learning objectives through text only or through game mechanics performed different results depending on the learning objective.Although usage of both text and game mechanics could reinforce certain learning objectives and hence improve learning, it is also possible that the usage of both text and game mechanics creates an extra cognitive load for the players which lead to overload for the students (Kiiili, 2005).

Figure 2 .
Figure 2. Pushing lactobacillus bacteria into a glass of milk results into the milk becoming yogurt

Figure 5 .Figure 6 .
Figure 5. "Bad bug" caught in a soap bubble just before disappearing

Figure 7 .
Figure 7. Percentage improvement in the knowledge of the players for the three cases: game mechanics only, text/narratives only and a combination of text/narratives and game mechanics