Mobile Health (mHealth) is a rapidly expanding area in health education and healthcare systems, and Kostkova succinctly presents the great potential of mHealth to impact the health education sector and future challenges facing the field (Kostkova, 2015). mHealth initiatives on mobile devices have demonstrated value for overcoming obstacles of rough terrain, remote populations, and limited resources to distribute important public health information to hard to reach populations (Martinez et al., 2017; Paudel et al., 2013; Raghu et al., 2015; Style et al., 2017). While a number of mHealth projects have been successfully implemented in clinical urban settings in LMIC (Birjovanu et al., 2019; Evans et al., 2021), mobile technology is of particular importance for health education and training for community workforce and local populations in rural and remote areas in low and middle income countries (LMIC). MANTRA is one of the first mobile educational projects co-developed with local women in Nepal. The “Maternal and Neonatal Technologies in Rural Areas (MANTRA): Increasing maternal and child health resilience before during and after disasters using mobile technology in Nepal” project investigated building women’s resilience by improving access to knowledge by developing mobile technology, a serious game, to support and expand existing participatory learning public health interventions and social protection mechanisms delivered by Female Community Health Volunteers (FCHVs) in a small area (usually about 1000 population) through regular women’s groups.
In rural and remote areas mHealth has the unique opportunity to deliver cost effective, easy to update /localise educational and training interventions (Mueller et al., 2019; Mueller et al., 2020a). In many LMIC countries, primary health systems utilize health volunteers who benefit from educational mHealth apps (Mueller et al., 2019), low skilled and junior rural doctors in need for mobile decision support tools (Gray et al., 2020), and health and environmental agents responsible for community surveillance (Aldosery et al., 2021). Evaluating these initiatives is a key step to demonstrate their value in achieving educational outcomes and appropriateness with the target audience in the health systems context.
This study aims to evaluate knowledge gain and retention from game analytics collected as women in Nepal played the MANTRA serious game. The game relies on players interpreting health conditions and risks from specifically designed pictograms of health conditions. The sequence of correct and incorrect answers as players move through increasingly difficult levels was logged in the application. Later analysis evaluated players’ knowledge gain and knowledge retention of the content in the game. This method of evaluation draws on game analytics to quantify the effectiveness of each learning objective to then improve the effectiveness of the game, thus delivering a comprehensive intervention to local rural communities.
Learning technology has made a major leap forward in the last decade. Serious and educational games, which are games with an educational purpose beyond entertainment, are now well established as a research domain (Klabbers, 2003; Kostkova, 2015). Learning effectiveness of serious games on learning has been demonstrated by knowledge assessment and evaluation (Farrell et al., 2011; Roozeboom et al., 2017; Tuli et al., 2017). However, how effective and impactful it actually is - remains the subject of ongoing research. In particular, knowledge assessment is an essential part of any educational experience, and serious games are no different (Iten & Petko, 2016).
Serious games are an important and as yet underutilized mHealth intervention opportunity to use game-based learning to supplement public health campaigns in LMIC settings (Borda et al., 2019; Borda et al., 2023; Kostkova, 2015). Learning objectives are specific pieces of information or concepts within the module that support the broad educational and training project goals, these are a form of a syllabus in traditional training (Jamieson & Grace, 2016; Molnar & Kostkova, 2013a, 2013b, 2018). These objectives create a clear goal for the intervention that can be quantitatively evaluated. Assessing serious educational games by evaluating learning through knowledge assessment is essential to quantify an educational experience. Traditionally in serious games, pre- and post-play assessments of knowledge take place outside the game as a test or a survey, in line with teaching and intervention assessment methods (Madle et al., 2004; Molnar & Kostkova, 2018; Roozeboom et al., 2017; Tuli et al., 2017).
However, the focus on traditionally established pre- and post assessment leaves the in-game collected interactive data and seamless assessment opportunities underused. In-app collected data provide an invaluable dataset for assessing performance and assessing user knowledge seamlessly without the participant knowing at every step (every click in fact) when interacting with the digital intervention (El-Nasr et al., 2013). The richness of the in-app collected dataset and insights these can bring through games analytics methods range better understanding of the impact of the mHealth educational intervention, feedback which parts of the games work and which did not work, leading to the opportunity to personalisate the experience and learning process for each learner. These could have unprecedented impact if games analytics have been applied widely - so far these have not been much explored (El-Nasr et al., 2013).
Taking a closer look at each player’s progression through a game allows the researcher a richer picture of the learning process, knowledge retention, and user interactions with the learning objectives and game itself. In support of the ultimate learning and knowledge retention goals of games-based learning, the usability, acceptability and cultural appropriateness of the game also impacts learning (Poong et al., 2016; Shorey et al., 2018; Smith et al., 2017).
To improve the learning effectiveness, immediate feedback to the participant if the answer is correct or incorrect in the game itself to 'reinforce learning' has been proven to increase knowledge gain (Molnar et al., 2015; Molnar & Kostkova, 2014). Further building on immediate feedback, the ‘reinforced learning’ approach has been also built into educational tools as a successful integration strengthening the knowledge gain with immediate knowledge assessment (Molnar et al., 2015; Molnar & Kostkova, 2014). Reinforced learning has been proven to increase the knowledge gain (Molnar 2012, Molnar 2018).
While MANTRA provides a unique example of such a successful educational mobile tool for low literacy workforce like FCHVs as well as rural women themselves, further research into the needs, challenges, and existing health systems in specific regions will ensure the technology is accepted by patients, health workers, communities, and healthcare systems (Kostkova, 2015; Sardi et al., 2017).
To study the effectiveness of the MANTRA serious game to retain health education knowledge, we developed a novel method to assess the impact of a serious game on FCHVs and rural women’s knowledge gain and retention using in-game collected session data and games analytics. This method evaluates the sequence of player responses collected in the game to construct a richer picture of the player experience and knowledge retention. To better understand how players gained and retained knowledge throughout the serious game intervention, we analysed how each player progressed through the game and the order of correct and incorrect responses for each learning objective. Analysing knowledge change and game-based learning identified successes and improvements is a novel approach easily generalisable for any mHealth education, training, or decision support systems, and will inform the next iteration of mobile games development, as well as insights that are transferable to similar research games projects in LMICs.
With these principles in mind, the interdisciplinary educational content of maternal, neonatal health, and geohazards were incorporated into MANTRA as a global health intervention. In the next section we illustrate on this unique study how a design of a successful game for such conditions could be evaluated using seamless evaluation and games analytics.