The research study aimed to develop a game-based version of the Test of Logical Thinking (TOLT) using the MDA framework. This study was conducted through a mixed methods approach with a quantitative emphasis and a nested qualitative approach [19]. The confidentiality of scores and feedback was ensured through anonymity, and candidates agreed to data sharing for information purposes, prior to taking part in the testing process.
3.1 Research Methodology
3.2 Phase 1: Adoption of the MDA framework
The first phase included the adoption of the MDA framework to design a continuous flow for the standard TOLT test. The approach suggested by Cheng et al. [6] has been utilized, wherein the pen-and-paper-based original TOLT test has been converted into a computerized game while retaining the original format and content, with minor changes being made in order to adapt the questions to a continuous flow. The game design is elaborated as follows: (1) Mechanics: A performance bar is displayed and updated at the end of each task, and Timing is measured per scenario. A question mark appears at the top of the screen whenever there is a new question. The player should be able to move past a question by clicking on the checkmark at the top of the screen. A retry option should be displayed for questions that allow it. An inventory is displayed at the top right of the screen. Mechanics are designed in a way that a novice game player would be able to navigate through it comfortably. (2) Dynamics: The player can move freely throughout the game and walk through the maze. The player can move forward in the scenarios when ready. There was a smooth flow of the questions related to the scenario and the reason. The player is expected to jump through and move around to access questions and to continue in the game. (3) Aesthetics: Visual appeal is achieved through color synchronization using a color palette. Immersion is provided through the nature of the different activities. The narrative of a scavenger hunt is maintained through question adaptation. Motivation to complete the test is enhanced through the performance bar and the nature of the scenarios.
The scenarios were as follows:
Scenario 1: (Fig. 1 and Fig. 2)
-
Screen: “Welcome to the Scavenger Hunt. You have been chosen for this quest to find a valuable treasure hidden by King W II 500 + years ago in the mysterious town, now turned into a deserted maze in Egypt”
-
Objective: Introduction to the task ahead: Give knowledge and set expectations of the setting, environment, and task.
-
Mechanics: Welcome Banner, the phrase and introduction to the task, Character appears, Performance Bar, Timer. The player is expected to click on it to proceed.
-
Dynamics: The player can click on the screen when they are ready to proceed at their own pace. The questions can be accessed at any time using the question mark on the screen. The player can click on the checkmark when they want to proceed to the next question.
-
Aesthetics: The initial feeling of curiosity and mysteriousness, bright colors for visual appeal.
Scenario 2: (Fig. 3 – Fig. 5)
-
Screen: “The overall maze covers 50km in 5 equal laps(parts). On average, players require 20 water bottles over the full course. You can only carry water bottles sufficient for each part at a time.” Collect water from rack. (Have a rack of water bottles to pick from, output number of bottles picked) “Why did you pick that many?”
-
Corresponding Question in the TOLT Test: Proportional Reasoning
-
Mechanics: Bottles get picked when the player steps on them. The player has to move and jump onto the table. As soon as he finishes picking, he can exit by clicking on the tick mark on the top. The inventory gets updated with numbers in real time
-
Dynamics: The player can move around and exit when wanted. The scenario can be restarted once if required.
-
Aesthetics: Movements such as jumping around, and colliding with the object is required
Scenario 3:
-
Screen: “In order to start the game you need to have 117 calories in your system. 1 packet of food contains 13 calories.” Collect food from rack. (Have a rack of food packets to choose from, output number of packets picked) “Why did you pick that many packets?”
-
Corresponding Question in the TOLT Test: Proportional Reasoning
-
Mechanics: Food packets get picked when the player steps on them. The player has to move and jump onto the table. As soon as he finishes picking, he can exit by clicking on the tick mark on the top. The inventory gets updated with numbers in real time
-
Dynamics: The player can move around and exit when wanted. The scenario can be restarted once if required.
-
Aesthetics: Movements such as jumping around, and colliding with the object are required.
Scenario 4: (Fig. 6 and Fig. 7)
-
Screen: Enter into the maze. (Lap 1) Pick protective armor. The box of armor has 21 suits: 3 short red suits, 4 short yellow suits, 5 short orange suits, 4 long red suits, 2 long yellow suits and 3 long orange suits. Click on the box and one suit randomly appears and avatar is dressed with it. Ask “What was the chance that you would have gotten a red suit?” “Why?”
-
Corresponding Question in the TOLT Test: Probabilistic Reasoning
-
Mechanics: The click on the box should trigger a random generator and the suit of the player should be automatically changed based on the chosen color.
-
Dynamics: The player is allowed to randomly pick using the button, thereby limiting player agency, as an outfit change needs to happen.
-
Aesthetics: The list of items should be readable and presented in a way that is not boring for the player
Scenario 5:
-
Screen: Walk into the maze. Include different objects in the environment. Allow him to move forward without any monitored questions.
-
Objective: Allow the player to familiarize with the environment and immerse in the scavenger hunt.
-
Mechanics: The player can move forward using the WASD keys, and can pick, and drop different objects around.
-
Dynamics: The player can move around as he wants.
-
Aesthetics: Promote immersiveness through the different items provided.
Scenario 6: (Fig. 8–10)
-
Screen: “There are four places in the total maze where you can place a torch, a sword, an extra water bottle, and an extra safety kit. Place them in these places as required.” “How many different combinations of placements can there be? “
-
Corresponding Question in the TOLT Test: Combinatorial Reasoning.
-
Mechanics: Multiple instructions given on how to pick, drop, and move items around in the playground provided to test the combinations possible. The player will need to interact and click to move items around.
-
Dynamics: The ease of moving items around should be facilitated, and the player needs to be able to exit when required and enter into the answer boxes, move back and forth between the playground and the answer grid.
-
Aesthetics: The enjoyment of moving items around and interacting with the different items.
Scenario 7:
-
Screen: Proceed through the maze. Can send them in different directions and around
-
Objective: Allow the player to familiarize with the environment and immerse in the scavenger hunt. Also, provide a small break for the player to answer questions.
-
Mechanics: The player can move forward using the WASD keys, can pick, and drop different objects around and interact with the objects.
-
Dynamics: The player can move around as he wants.
-
Aesthetics: Promote immersiveness through the different items provided and the backgrounds and environments.
Scenario 8:
“This part of the maze has been flooded recently. There is a set of ropes of different lengths hanging from the ceiling with buckets of different weights tied to the bottom end of them. The weight of each bucket is shown near each. This is given in Fig. 11”
On average, it takes two minutes to cross this part of the maze. “You are allowed to experiment to find out if changing the length of these would change the amount of time to swing back and forth. What would you use?” Give answers to choose from. Ask why?
“You are allowed to experiment to find out if changing the weight of these would change the amount of time to swing back and forth. What would you use? “Give answers to choose from. Ask why? Allow to jump onto them one by one and move to the ground at the end of it. (If he falls down, allow to retry)
-
Corresponding Question in the TOLT Test: Controlling Variables
-
Mechanics: The player should be able to jump from rope to rope, which is a simulation of jumping onto them to cross the waterway. The user should have access to this diagram for reference when answering the question. If they fail to jump using the ropes and keep falling, they should also have an exit option to go directly to the question.
-
Dynamics: The exit paths are defined so that they do not need to spend too much time retrying if jumping fails.
-
Aesthetics: An immersive experience provided for the player to get an understanding of the movement referred to in the question, as shown in Fig. 12 and Fig. 13.
Scenario 9:
-
Screen: There is a box of food containing three sandwiches and three pizzas. Press button, and one item is chosen randomly. “What is the chance that it could have been a pizza?” Give answers to choose from. Why? Let him sit and eat and proceed.
-
Corresponding Question in the TOLT Test: Probabilistic Reasoning.
-
Mechanics: The user should be able to press the button to choose, and a random selector should generate the item, as shown in Fig. 14.
-
Dynamics: The player is not able to choose from the food items; instead, a random generator provides one item.
-
Aesthetics: Pressing the button to generate immersiveness, sitting, and eating to generate the actual feeling of being in the scavenger hunt.
Scenario 10:
-
Screen: “You can choose 3 support creatures to join you in the mission from this stage onwards.”
-
There are 3 groups: Rats, Rabbits and Humans. You can choose 1 from each group.
-
Rats: Tom (T), Sally (S), Bill (B)
-
Rabbits: Jerry (J), Anne (A), Connie (C)
-
Humans: Dan (D), Martha (M), Gwen(G)
What are the possible combinations? (Give boxes to place combinations - there should be 3*10 spaces given) How many combinations are there? Allow to proceed when they are done. (Even if the required number of combinations is not reached.
-
Corresponding Question in the TOLT Test: Combinatorial Reasoning.
-
Mechanics: The player should be able to view an overall picture of the part of the map he is in, with a set of rabbits, rats, and humans to choose from. The entering of the combinations should be arranged in a way that allows input of three letters, as given in Fig. 15, and the accuracy of the entries should be checked for no repetitions of the combinations.
-
Dynamics: They should be able to exit the stage whenever required even if all the combinations have not been entered. They should be able to view the list of choices on the screen where they enter the combinations.
-
Aesthetics: Immersiveness and interactivity are generated through the nature of the task, and visual appeal is generated through adding in creatures.
Scenario 11:
Screen: Door opens and there is a waterfall that falls into a pond. There are flowers around it and birds and animals. There are fish in the pond (clearly visible) He can zoom into them and observe them through an image. Are fat fish more likely to have broad stripes than thin fish? Why? If they choose yes or no, open two separate doors that lead them to another door through different paths.
-
Corresponding Question in the TOLT Test: Correlational Reasoning.
-
Mechanics: The player should be able to walk around, as given in Fig. 17, find where the fish are, and view the original image as a summary before they answer. The question appears when they find the fish and collide with them.
-
Dynamics: The player can move freely and explore around. He can zoom on the image, as given in Fig. 16 and Fig. 18 (summary) in the answering screen.
-
Aesthetics: The colors of the fish for visual appeal, immersiveness, and enjoyment
Scenario 12:
-
Screen: There is a rusty environment with old sacks. “There are 30 dead mice in the room. Collect them in one place.” This is shown in Fig. 20.
-
“Are fat mice likely to have black tails and thin mice more likely to have white tails?” Yes or No? Why?
-
Corresponding Question in the TOLT Test: Correlational Reasoning.
-
Mechanics: The player should be able to walk around, find where the mice are, and also to view the original image as a summary before they answer. They can proceed to the question when they have finished picking up all the rats. The inventory should have an updated number of mice picked up each time.
-
Dynamics: The player can move freely and explore around. He can zoom in on the image given in Fig. 19 (summary) on the answering screen.
-
Aesthetics: The nature of the mice getting picked should create enjoyment, enthusiasm, and immersiveness.
Scenario 13:
-
Screen: Results Screen
-
Objective: Displaying the results of the test right away so that the player walks out with feedback on performance.
-
Mechanics: The correct/incorrectness of each question should be displayed, and the categorical score should be displayed along with the time taken.
-
Dynamics: The user should be able to navigate through the results sheet to get a sense of performance, but not to go back to the game and refer to the questions.
-
Aesthetics: The information should be presented in a visually appealing manner.
To develop the prototype of the game-based version of the created scenarios a comprehensive 3D game design methodology was used through the Unity Game Engine. PlayFab was used as the backend, which is a platform providing specialized tools and services that support the development of video games.
The developed game-based TOLT test was tested using a focus group of eight freshly recruited entry-level graduate software engineers. They took both the paper-based, original TOLT test and the newly developed game-based TOLT test, and their scores in both tests were compared to assess the success of adapting the questions and provide the candidates with a holistic perspective for comparison.
3.3 Phase 2: Game development
To develop the prototype version of the game-based version of the created scenarios, a comprehensive 3D game design methodology was used through the Unity Game Engine. PlayFab was used as the backend, which is a platform providing specialized tools and services that support the development of video games.
3.4 Phase 3: Administration of the test
The developed game-based TOLT test was tested using a focus group of eight freshly recruited entry-level graduate software engineers. They took both the paper-based original TOLT test and the newly developed game-based TOLT test, and their scores in both tests were compared to assess the success of adapting the questions and provide the candidates with a holistic perspective of comparison.
3.5 Phase 4: Obtaining feedback on candidates’ perceptions of the two modes
The game-based TOLT test was evaluated based on the following metrics related to the adapted MDA Architecture. These metrics were derived from the studies of Trinidad et al., Hunicke et al., and Mora et al. [9, 11, 13]. The candidates provided ratings for these metrics based on an assigned Likert Scale, providing quantitative data for analysis, in addition to qualitative feedback.
They were as follows: (1) Mechanics included complexity (rules, actions, or components in the game), interactivity (interactive elements or objects in the game), balance (the fairness and equilibrium of the game mechanics), skill requirements (evaluating the level of skill needed to use and proceed through different mechanics), and progression (analyzing the progression of mechanics throughout the game). (2) Dynamics involved flow (the smoothness and continuity of player actions and feedback), challenge (the difficulty level and progression of challenges in the game), pace (the speed and rhythm of gameplay), player agency (examining the degree of player freedom and influence over the game world), and emergent behavior (identifying unexpected or unintended interactions between game elements). (3) Aesthetics included immersion (evaluating the level of player engagement and presence in the game world), narrative (assessing the quality and impact of the game's storytelling elements), visual appeal (measuring the graphical quality, art style, and overall visual experience), enjoyment, motivation to complete the test, attention span during the test, and reducing test-taking anxiety. (4) Overall feedback involved questions such as Do you play video games and other e-games? (All the time, Regularly, Often, Rarely, Never), impact on positive impression about a potential employer, Would they pick this employer over a competitor who has all other factors being the same, but has a traditional recruitment testing method? Any other feedback/thoughts?