The Effectiveness of Intelligent Tutoring Systems in Supporting Students with Varying Levels of Programming Experience

doi:10.21203/rs.3.rs-4788797/v1

Download PDF

Research Article

The Effectiveness of Intelligent Tutoring Systems in Supporting Students with Varying Levels of Programming Experience

https://doi.org/10.21203/rs.3.rs-4788797/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This study investigates the effectiveness of Intelligent Tutoring Systems (ITS) in supporting students with varying levels of programming experience. Through a mixed-methods research design, the study explores the impact of ITS on student performance, adaptability to different skill levels, and best practices for utilizing ITS in heterogeneous programming classrooms. The quantitative analysis reveals significant differences in the influence of ITS on improving programming skills, the usefulness of ITS feedback, and satisfaction levels with the interface based on students' programming experience levels. Advanced students show the most significant improvement in programming skills and find the feedback more valuable, while intermediate and advanced students report higher satisfaction with the ITS interface. The qualitative analysis highlights positive impacts on the learning process, challenges faced, and suggestions for improvement. This study contributes to the literature by emphasizing the importance of tailoring ITS interventions to meet the unique needs of students with varying levels of programming experience, offering insights for educators, curriculum developers, and educational technology designers.

Educational Technology

Intelligent Tutoring Systems

Heterogeneous Learning Environments

Personalized Instruction

Programming Education

Student Proficiency Levels

Intelligent tutoring systems (ITS) have been increasingly utilized in educational settings as they can effectively diagnose knowledge gaps and misconceptions (Moraes & Pedreira, 2020). These systems have evolved to accommodate many users and are particularly valuable in e-learning environments (Novickis & Rikure, 2005; Bradác & Kostolanyova, 2016). Semantic-net principles in these systems have been shown to improve query precision and user interface (Guang, 2003). Furthermore, integrating teaching activity trees in intelligent tutoring systems allows for personalized education and easier monitoring of student progress (Zhang et al., 2010).

The emergence of ITS in programming education has been a significant development, with a focus on simplifying the learning process for novice programmers (Al-Jumeily et al., 2014). These systems have the potential to provide automated feedback and grading, addressing the limitations of existing systems (Fan et al., 2023). A systematic review of ITS in programming education highlights the variety of systems available, each with its unique features (Crow et al., 2018). The use of Bayesian networks in a web-based ITS for computer programming further demonstrates the potential for these systems to guide students through online course materials and recommend learning goals (Butz et al., 2004). Prior research has explored the use of ITS in programming education and also highlights its potential benefits in personalized instruction and skill development. Dadic (2011) emphasizes the importance of knowledge representation and semantic analysis in these systems, which enable them to adapt to individual student needs. Khazanchi and Khazanchi (2021) further underscore the role of artificial intelligence in providing adaptive instructions, particularly for students with disabilities. Conati and Kardan (2013) extend this discussion by highlighting the use of student models to support personalization in a range of educational activities, including problem-solving and exploratory open-ended tasks.

However, many studies have focused on homogeneous groups of students or lacked a comprehensive examination of how ITS can adapt to diverse programming experience levels. This gap in the literature calls for further investigation into the effectiveness of ITS in supporting a range of student skill levels in programming. While some students may have prior experience or advanced skills in programming, others may be novices entering the field. Addressing these differences in programming backgrounds is crucial for maximizing the impact of ITS on learning outcomes and ensuring equity in educational support.

This study aims to investigate the effectiveness of intelligent tutoring systems in supporting students with varying levels of programming experience. Specifically, this study's research question is: How can intelligent tutoring systems be optimized to enhance learning outcomes for students with diverse programming backgrounds?

The objectives of this research include

Assessing the impact of ITS on student’s performance
Examining the adaptability of ITS to different skill levels and
Identifying best practices for utilizing ITS in heterogeneous programming classrooms

Understanding how it can effectively support students with varying programming experience levels is crucial for advancing programming education and promoting student success. By investigating the factors that contribute to the effectiveness of ITS in heterogeneous learning environments, this study provides valuable insights for educators, curriculum developers, and educational technology designers.

The paper is structured as follows: The next section reviews relevant literature on intelligent tutoring systems and programming education. Section three presents the theoretical framework of the study. Subsequently, the methodology is outlined, detailing the research design and data collection methods. The study results are then presented and discussed about the research question. Finally, the conclusions from the study's findings and recommendations for future research and educational practice are discussed.

This part focuses on using Intelligent Tutoring Systems in education, specifically focusing on programming education. The review highlights the increasing importance of ITS in enhancing learning outcomes, providing personalized instruction, and catering to diverse user needs. Previous studies have demonstrated the effectiveness of ITS in improving student performance and guiding learning experiences in a variety of settings. However, challenges such as the need for more in-depth explanations of AI techniques and interoperability issues remain areas of concern in the field. This section provides a comprehensive overview of the existing literature on ITS in programming education, emphasizing its potential benefits and current limitations.

2.1. Overview of Intelligent Tutoring Systems in Education

The current body of literature discusses using ITS in education from various perspectives. Novickis and Rikure (2005) highlight the increasing importance of intelligent tutoring systems, particularly in tertiary education, and their adaptability to various users. Hoppe et al. (2021) present the use of an intelligent tutoring system as a formative assessment tool in design education, demonstrating its ability to guide students to valid solutions through supportive feedback. Intelligent Tutoring Systems have been increasingly utilized in education to enhance learning experiences. Hoppe et al. (2021) and Epstein et al. (2013) both highlight the effectiveness of ITS in providing formative assessment and personalized instruction, respectively. Guang (2003) and Smith et al. (1987) discuss the technical and design aspects of ITS, emphasizing the need for user-friendly interfaces and the integration of human factors. Furthermore, Šarić-Grgić et al. (2019) provide a comprehensive review of ITS, focusing on using intelligent agents in their architecture and the shift towards adaptive instructional systems.

ITS are crucial for students and significantly impact their achievement (Swanson, 2019). These systems, which use adaptivity to cater to a wide range of users, have the potential to enhance the learning experience significantly. They can improve query precision and user interface in web-based education (Guang, 2003) and are seen as a critical component of knowledge-based expert systems (Sokolnicki, 1991). The development of ITS is a crucial area of research, with a focus on knowledge representation, student modeling, planning, natural language issues, explanations, and learning (Sokolnicki, 1991). Moreover, Intelligent Tutoring Systems through personalized e-learning are more effective than traditional instruction methods, particularly when combined with on-screen agents and personalization (Swanson, 2019; Rokade & Jawandhiya, 2019). They have also been shown to be beneficial in programming teaching, where they can provide automated feedback and grading (Fan et al., 2023). Furthermore, ITS can be tailored to individual student needs, considering their behavior and emotions, to enhance the teaching-learning process (Rathore & Arjaria, 2020).

2.2. Discussing previous research on using ITS in programming education

A range of studies have explored using Intelligent Tutoring Systems in programming education. Gavrilovic and Ovanovićo (2015) and Lee and Baba (2005) both highlight the potential of ITS in addressing the challenges of teaching programming, with Gavrilovic and Ovanovićo specifically discussing the implementation of ITS for Java programming e-learning. Schez-Sobrino et al. (2020) and Xhakaj and Aleven (2018) further contribute to this discussion by proposing dynamic graphic visualizations and conceptually oriented activities to enhance the learning experience. These studies collectively underscore the potential of ITS in improving the effectiveness and accessibility of programming education.

However, challenges remain in the field, including the need for more in-depth explanations of the relationship between AI techniques and ITS data (Francisco & Silva, 2022). The effectiveness of ITS in delivering different types of knowledge and guiding learning experiences has been highlighted, particularly in the Science, Technology, Engineering, and Mathematics (STEM) domains (Feng et al., 2021). Furthermore, the interoperability of ITS for programming, particularly in the context of programming exercises, has been identified as a significant issue (Hieu, 2019).

In sum, the literature review underscores the significant impact of Intelligent Tutoring Systems on learning outcomes and the teaching process, particularly in programming education. Previous research has highlighted the potential of ITS to enhance the effectiveness and accessibility of programming instruction through personalized feedback and adaptive learning approaches. However, challenges such as the need for more detailed explanations of AI techniques and interoperability issues pose barriers to the widespread implementation of ITS in programming education. This study seeks to address these limitations by examining the optimization of ITS to support students with varying levels of programming experience, contributing to the advancement of best practices for utilizing ITS in heterogeneous programming classrooms.

Cognitive Load Theory (CLT) provides a valuable theoretical framework for understanding how Intelligent Tutoring Systems can support student learning in programming education. CLT posits that cognitive load, or the mental effort required to process information, plays a crucial role in learning and problem-solving (Sweller, 1994). According to CLT, learners have a limited capacity to process information, and cognitive overload can hinder learning by inhibiting the acquisition and retention of new knowledge (Sweller, 2006).

In programming education, students must often juggle multiple cognitive tasks, such as understanding coding syntax, problem-solving logic, and debugging errors, leading to a high cognitive load. High cognitive load in programming education can present significant challenges for students, particularly novices (Renumol et al., 2009; Yousoof et al., 2014). It can lead to rote learning and memorization, hindering the development of a deeper understanding of programming concepts (Yousoof & Sapiyan, 2016). ITS has been shown to mitigate cognitive overload in programming students. Lamia et al. (2018) found that an ITS combined with a social network improved problem-solving performance. Müller et al. (2018) explored the feasibility of using a cognitive system, IBM Watson, as a virtual tutor in an introductory programming course, indicating potential for future use. Orhun (1991) discussed the importance of knowledge representation in ITS for computer programming, a critical factor in reducing cognitive load. Yousoof et al. (2007) proposed a mechanism to measure and manage cognitive load in programming instruction, which could be incorporated into ITS to enhance learning.

By applying Cognitive Load Theory to the study of ITS in programming education, this research seeks to investigate how ITS can optimize cognitive load management for students with varying levels of programming experience. The study aims to identify strategies for reducing extraneous cognitive load, optimizing germane cognitive load, and promoting effective learning techniques to enhance student engagement and skill development in programming tasks. By leveraging the principles of CLT within the design and implementation of ITS, this study aims to contribute valuable insights to the field of programming education and support the effective use of technology-enhanced learning environments in fostering student success.

The study utilized a mixed-methods research design (William, 2024) to investigate the effectiveness of Intelligent Tutoring Systems in supporting students with varying levels of programming. This research aims to explore the effectiveness of ITS in supporting student learning and skill development in programming tasks. The epistemological stance adopted for this study is pragmatism, which acknowledges the value of both quantitative and qualitative data in understanding complex phenomena and informing educational practices (Creswell, 2014).

The target population for this study consists of 160 computer science students from a university in Southern China who have previous experience with ITS in programming education. A purposive sampling approach is used to select participants who have engaged with ITS in their learning process. The study includes a diverse group of participants to capture a range of perspectives and experiences with ITS. Figure 1 below is one of the ITS the respondents use in programming.

Data are collected through a questionnaire containing a combination of closed-ended and open-ended questions. The closed-ended questions assess students' perceptions of the effectiveness, usability, and impact of ITS on their learning outcomes. The open-ended questions allow participants to elaborate on their experiences, challenges faced, and suggestions for improvement regarding the ITS in programming education.

Quantitative data analysis involves descriptive statistics summarizing participants' responses to closed-ended questions. Comparative analyses, such as chi-tests and ANOVAs, examine potential differences in perceptions and outcomes based on students' programming experience levels. Qualitative data from open-ended responses are analyzed using thematic analysis (William, 2024) to identify common themes and patterns in students' feedback on their experiences with the ITS. In addition, NVivo 14 is employed for qualitative data analysis. Simultaneously, the research findings are thoroughly and comprehensively analyzed using IBM Statistical Package for the Social Sciences (SPSS) Statistics 29.0.2.0, guaranteeing a solid and comprehensive quantitative analysis. Table 1 below displays the respondents’ traits.

Table 1

Respondents’ traits
Characteristics	Options	Frequency	Percentage
Sex	Male	136	85
	Female	24	15
	Other	0	0
Level of education	Bachelor	160	100
	Master	0	0
	Ph.D.	0	0
Age	Under 18	0	0
	18–24	152	95
	25–34	8	5
	35–44	0	0
Level in programming	Beginner	136	85
	Intermediate	16	10
	Advanced	8	5

This section presents and discusses the results of the quantitative and qualitative analysis conducted to explore the impact of Intelligent Tutoring Systems on students with varying levels of programming experience. Through chi-square and ANOVA tests, the quantitative analysis examines associations between programming proficiency levels and the effectiveness of ITS. The qualitative analysis highlights themes related to the positive impact of ITS, challenges faced by students, and suggestions for improvement. By integrating these findings, this section offers valuable insights into optimizing ITS interventions for students with diverse programming backgrounds.

5.1. Quantitative analysis

Results of chi-square tests

This study performed chi-square tests to identify potential associations between students' level in programming and the increase in their confidence and motivation in programming since using ITS on the one hand. On the other hand, chi-square tests were performed to check potential associations between students' level of programming and the usefulness of ITS in completing programming tasks. Results are displayed in Tables 2 and 3 below.

Table 2

Chi-Square Tests 1
	Value	df	Asymptotic Significance (2-sided)
Pearson Chi-Square	4.983^a	2	.083
Likelihood Ratio	8.515	2	.014
Linear-by-Linear Association	4.317	1	.038
N of Valid Cases	160

a. 2 cells (33.3%) have expected count less than 5. The minimum expected count is 1.20.

Table 3

Chi-Square Tests 2
	Value	df	Asymptotic Significance (2-sided)
Pearson Chi-Square	1.486^a	2	.476
Likelihood Ratio	2.674	2	.263
Linear-by-Linear Association	1.287	1	.257
N of Valid Cases	160

a. 2 cells (33.3%) have expected count less than 5. The minimum expected count is .40.

The chi-square is 4.983, with p-values of 0.083. Given that the p-value is higher than 0.05, there is a non-significant association between students' level in programming and the increase in their confidence and motivation in programming since using ITS. In other words, there is insufficient evidence to claim a significant association between students' level in programming and the increase in their confidence and motivation in programming since using ITS. The higher p-value indicates that the difference in confidence and motivation levels between different levels of programming students may be due to random chance rather than an actual relationship.

Similarly, the following chi-square is 1.486, with p-values of 0.476. Given that the p-value is higher than 0.05, there is a non-significant association between students' level of programming and the usefulness of ITS in completing programming tasks. Put differently, findings suggest that there is not enough evidence to suggest a significant association between students' level of programming and the usefulness of ITS in completing programming tasks. The higher p-value indicates that the difference in the perceived usefulness of ITS in completing programming tasks among different levels of programming students may be due to random chance rather than an actual relationship.

ANOVA test results

We conducted three ANOVA tests regarding the influence of ITS in improving programming skills, the usefulness of ITS feedback on programming tasks, and the satisfaction with using the interface and functionality of the ITS. Tables 4 and 5 show the result of the first ANOVA test.

Table 4

ANOVA 1: Influence of ITS in improving programming skills
	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	5.365	2	2.682	5.524	.005
Within Groups	76.235	157	.486
Total	81.600	159

Table 5

Multiple Comparisons 1: Post Hoc Tests
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	95% Confidence Interval
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	Lower Bound	Upper Bound
Beginner	Intermediate	.294	.184	.250	− .14	.73
Beginner	Advanced	− .706^*	.254	.016	-1.31	− .11
Intermediate	Beginner	− .294	.184	.250	− .73	.14
Intermediate	Advanced	-1.000^*	.302	.003	-1.71	− .29
Advanced	Beginner	.706^*	.254	.016	.11	1.31
Advanced	Intermediate	1.000^*	.302	.003	.29	1.71

*. The mean difference is significant at the 0.05 level.

The significance level of 0.005 in Table 4 is lower than the set level of 0.05. Therefore, there is a statistically significant difference in the influence of ITS in improving programming skills among students of different levels of programming experience, with the advanced students showing the most significant improvement compared to beginner and intermediate students. The post hoc tests offer ample details about differences, notably between "Beginner" and "Advanced" and between "Intermediate" and "Advanced," with significance levels of 0.16 and 0.003, respectively. Students' proficiency in programming may impact the influence of ITS in improving their programming skills.

Tables 6 and 7 display the results of the second ANOVA test.

Table 6

ANOVA 2: Usefulness of ITS feedback on programming tasks
	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	5.835	2	2.918	3.476	.033
Within Groups	131.765	157	.839
Total	137.600	159

Table 7

Multiple Comparisons 2: Post Hoc Tests
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	95% Confidence Interval
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	Lower Bound	Upper Bound
Beginner	Intermediate	.176	.242	.747	− .40	.75
Beginner	Advanced	− .824^*	.333	.038	-1.61	− .03
Intermediate	Beginner	− .176	.242	.747	− .75	.40
Intermediate	Advanced	-1.000^*	.397	.034	-1.94	− .06
Advanced	Beginner	.824^*	.333	.038	.03	1.61
Advanced	Intermediate	1.000^*	.397	.034	.06	1.94

*. The mean difference is significant at the 0.05 level.

The ANOVA test results for the usefulness of ITS feedback on programming tasks show a significant difference in the effectiveness of feedback based on the student's level of programming (F(2, 157) = 3.476, p = .033). The between-groups variance is 5.835, while the within-groups variance is 131.765, indicating a significant difference in mean scores between at least two of the three levels of programming.

The post-hoc tests further reveal the mean differences between the different levels of programming. The results indicate significant differences in mean scores between the beginner and advanced groups (Mean Difference = -0.824, p = .038) and intermediate and advanced groups (Mean Difference = -1.000, p = .034). At the same time, there are no significant differences between the beginner and intermediate groups (Mean Difference = 0.176, p = .747).

In summary, these results suggest significant differences in the usefulness of ITS feedback on programming tasks among students of different levels of programming experience. The feedback is more effective for advanced students than for beginner and intermediate students. This information can be valuable for educators and instructional designers aiming to enhance the effectiveness of ITS feedback in programming education.

Tables 8 and 9 present the results of the third ANOVA test.

Table 8

ANOVA 3: Satisfaction with using the interface and functionality of the ITS
	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	5.718	2	2.859	6.813	.001
Within Groups	65.882	157	.420
Total	71.600	159

Table 9

Multiple Comparisons 3: Post Hoc Tests
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	95% Confidence Interval
(I) Level in programming	(J) Level in programming	Mean Difference (I-J)	Std. Error	Sig.	Lower Bound	Upper Bound
Beginner	Intermediate	.471^*	.171	.018	.07	.88
Beginner	Advanced	− .529	.236	.067	-1.09	.03
Intermediate	Beginner	− .471^*	.171	.018	− .88	− .07
Intermediate	Advanced	-1.000^*	.281	.001	-1.66	− .34
Advanced	Beginner	.529	.236	.067	− .03	1.09
Advanced	Intermediate	1.000^*	.281	.001	.34	1.66

*. The mean difference is significant at the 0.05 level.

The ANOVA test results for satisfaction with using the interface and functionality of the ITS show a significant difference in satisfaction levels based on the student's level of programming (F(2, 157) = 6.813, p = .001). The between-groups variance is 5.718, while the within-groups variance is 65.882, indicating a significant difference in mean scores between at least two of the three levels of programming.

The post-hoc tests (multiple comparisons) reveal the mean differences between the different levels of programming. The results indicate significant differences in mean scores between the beginner and intermediate groups (Mean Difference = 0.471, p = .018) and intermediate and advanced groups (Mean Difference = 1.000, p = .001). At the same time, there was no significant difference between the beginner and advanced groups (Mean Difference = -0.529, p = .067).

In conclusion, these findings suggest significant differences in satisfaction levels with using the interface and functionality of the ITS among students of different levels of programming experience. Intermediate and advanced students show higher levels of satisfaction compared to beginner students. This information can be helpful for developers and educators looking to tailor the ITS interface and functionality to different levels of programming expertise to improve user satisfaction.

5.2. Qualitative analysis: Thematic analysis

Three themes emerged from the thematic analysis: positive impact on the learning process, challenges faced, and suggestions for improvement. This analysis offers a comprehensive understanding of the potential benefits and limitations of ITS in supporting students with varying levels of programming experience.

Positive impact on the learning process

Many respondents highlighted the positive influence of the Intelligent Tutoring System on their learning process. For example, one participant stated, "ITS helped me understand the topics I did not understand in the classes," while another mentioned, "It makes me calmer." These responses indicate that the ITS has aided comprehension and reduced anxiety during the learning process.

Challenges faced

While some participants reported no significant challenges, others mentioned specific difficulties. For instance, one respondent stated, "Sometimes I cannot find a course on the topic I want." At the same time, another mentioned, "Having restrictions on programming languages, tools, or project types may limit the scope and depth of my learning." These responses suggest that there may be limitations or barriers that impact the effectiveness of the ITS in programming education.

Suggestions for improvement

Participants offered suggestions for enhancing the ITS to support their programming education needs better. For example, one respondent recommended, "There should be more cooperation with Intelligent Agent (IA)," while another suggested, "Provides diverse learning resources such as video tutorials, interactive experiments, and programming challenges." These suggestions highlight the importance of incorporating additional features and resources to optimize the ITS for varied learning preferences and requirements.

The thematic analysis of the responses indicates that while the ITS positively impacts the learning process for many students, there are also challenges and areas for improvement that should be addressed to enhance its effectiveness in supporting programming education.

5.3. Integration of the findings

Integrating the quantitative and qualitative findings provides a comprehensive understanding of the effectiveness of Intelligent Tutoring Systems in supporting students with varying levels of programming experience. The quantitative analysis revealed significant differences in the influence of ITS on improving programming skills, the usefulness of ITS feedback on programming tasks, and satisfaction levels with the interface and functionality of the ITS based on students' programming experience levels.

The qualitative analysis further enriched these findings by highlighting the positive impact of ITS on the learning process, challenges faced by students, and suggestions for improvement. The qualitative data shed light on the benefits of ITS in aiding comprehension and reducing anxiety, as well as the limitations and barriers students may encounter when utilizing ITS in programming education. Additionally, students' suggestions for enhancing the ITS to meet their learning needs better emphasized the importance of incorporating diverse learning resources and enhancing collaboration with Intelligent Agents.

Overall, integrating quantitative and qualitative findings underscores the importance of optimizing Intelligent Tutoring Systems to cater to students' diverse programming backgrounds and preferences.

5.4. Discussions

Building on prior research findings

The findings of the quantitative analysis align with previous research indicating the effectiveness of Intelligent Tutoring Systems in programming education. Studies like those by Swanson (2019) and Fan et al. (2023) have highlighted the benefits of personalized e-learning and automated feedback in enhancing the learning experience. The current study builds on this foundation by examining the impact of ITS on students with varying levels of programming experience, shedding light on how proficiency levels can influence the effectiveness of ITS feedback and satisfaction.

Deviation from anterior research trends

In line with the studies by Gavrilovic and Ovanovićo (2015) and Lee and Baba (2005), our research deviates by uncovering the nuanced differences in the influence of ITS based on students' programming proficiency levels. Contrary to previous assumptions about ITS efficacy being consistent across all learners, our findings suggest that the effectiveness of ITS feedback and satisfaction levels varied significantly among beginner, intermediate, and advanced programming students. This departure from the general trend underscores the importance of considering students' proficiency levels in designing and implementing ITS in programming education.

Novelty of the study

The novelty of this study lies in its focus on the impact of ITS on students with diverse programming backgrounds. By conducting chi-square and ANOVA tests to examine the association between programming proficiency levels and the effectiveness of ITS, the study offers valuable insights into how individual student characteristics can influence the outcomes of ITS interventions. Furthermore, the qualitative thematic analysis highlights the specific challenges and suggestions for improvement provided by students, adding a qualitative dimension to the existing literature on ITS in programming education. Overall, the study contributes to the growing body of research on ITS by emphasizing the importance of tailoring ITS interventions to meet the unique needs of students with varying levels of programming experience.

This study investigated the effectiveness of Intelligent Tutoring Systems in supporting students with varying levels of programming experience. The findings from the mixed-methods research design provided valuable insights into the impact of ITS on student learning outcomes and the adaptability of ITS to different skill levels. As a result, best practices for utilizing ITS in heterogeneous programming classrooms are discussed below.

The quantitative analysis revealed significant differences in the influence of ITS on improving programming skills, the usefulness of ITS feedback on programming tasks, and satisfaction levels with the interface and functionality of the ITS based on students' programming experience levels. Advanced students showed the most significant improvement in programming skills and found the feedback more advantageous than beginner and intermediate students. Additionally, intermediate and advanced students reported higher satisfaction with the ITS interface and functionality.

The qualitative analysis highlighted three key themes: the positive impact of ITS on the learning process, challenges faced by students, and suggestions for improvement. Many participants reported that the ITS helped them understand topics and reduced anxiety, while others mentioned difficulty finding specific courses or limitations in the scope of learning. Suggestions for improvement included enhancing collaboration with Intelligent Agents and providing diverse learning resources.

6.1. Theoretical contributions

This study contributes to the literature by emphasizing the importance of tailoring ITS interventions to meet the unique needs of students with varying levels of programming experience. The findings suggest that individual student characteristics, such as programming proficiency levels, can significantly influence the effectiveness of ITS feedback and satisfaction levels in programming education.

6.2. Practical implications

The research findings indicate several best practices for utilizing Intelligent Tutoring Systems in heterogeneous programming classrooms.

First, it is crucial to implement adaptive learning strategies within the ITS to personalize the learning experience for students with varying levels of programming proficiency. It could involve adjusting the difficulty level of tasks, providing tailored feedback, and offering individualized learning pathways to meet the unique needs of each student.
Additionally, incorporating a variety of learning resources and activities within the ITS, such as interactive tutorials, coding challenges, and peer collaboration opportunities, can cater to different learning styles and preferences among students.
Furthermore, creating a supportive and collaborative learning environment by integrating Intelligent Agents within the ITS to provide timely feedback, guidance, and assistance can enhance students' engagement and motivation in learning programming concepts.
Lastly, continuous monitoring and evaluation of the ITS's effectiveness, as well as seeking feedback from students to identify areas for improvement, can contribute to the ongoing optimization of the learning experience in heterogeneous programming classrooms.

Educators and instructional designers can use the insights from this study to optimize ITS interventions for students with diverse programming backgrounds. By addressing challenges and incorporating suggestions for improvement, educators can enhance students' learning experience and outcomes using ITS in programming education. Tailoring ITS to individual student needs can lead to more personalized and effective learning environments.

6.3. Limitations and Prospective Research Opportunities

One limitation of this study is the potential for self-report bias in the student data. Future research could include objective measures of programming performance to complement self-reported data and validate findings. Further exploration of the impact of additional factors, such as gender, on ITS effectiveness could provide deeper insights into personalized learning approaches in programming education.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the author(s) used Grammarly AI to proofread and improve the manuscript’s language. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Acknowledgment

The author(s) is(are) grateful to E. K. and F. A. R. for their assistance in the data collection. The author (s) also thank(s) everyone, particularly students, who contributed to the writing of this paper through their participation in its survey.

Ethics Statement

The ethics waiver was approved by Sanya University

Ethical Approval

Not applicable

Informed Consent

Not applicable

Statement Regarding Research Involving Human Participants and/or Animals

Not applicable

Consent to Participate

Informed consent was obtained from the participants in the study.

Consent to Publish

Not applicable

Funding

This research received no external funding.

Author's Contribution

Conceptualization, AWFK; methodology, AWFK; software, AWFK; validation, AWFK; formal analysis, AWFK; investigation, AWFK; resources, AWFK; data curation, AWFK; writing—original draft preparation, AWFK; writing—review and editing, AWFK; visualization, AWFK; supervision, AWFK; project administration, AWFK; funding acquisition, AWFK

The author(all authors) has(have) read and agreed to the published version of the manuscript.

Competing Interests

The author(s) declare(s) no competing interests.

Availability of data and materials

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Al-Jumeily, D., Hussain, A.J., Alghamdi, M.I., Lamb, D.J., & Hamdan, H. (2014). The development of an intelligent tutorial system for system development. 2014 International Conference on Web and Open Access to Learning (ICWOAL), 1–6.
Bradác, V., & Kostolanyova, K. (2016). Intelligent Tutoring Systems. Journal of Intelligent Systems, 26, 717–727.
Butz, C.J., Hua, S., & Maguire, R.B. (2004). A Web-Based Intelligent Tutoring System for Computer Programming. IEEE/WIC/ACM International Conference on Web Intelligence (WI'04), 159–165.
Conati, C., & Kardan, S. (2013). Student Modeling: Supporting Personalized Instruction, from Problem Solving to Exploratory Open Ended Activities. AI Mag., 34, 13–26.
Creswell, J. W. (2014). Research design: qualitative, quantitative, and mixed methods approaches. 4th edn. United States of America: SAGE Publications, Inc.
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018, January). Intelligent tutoring systems for programming education: a systematic review. In Proceedings of the 20th Australasian Computing Education Conference (pp. 53–62).
Dadic, T. (2011). Intelligent tutoring system for learning programming. In Intelligent tutoring systems in e-learning environments: Design, implementation and evaluation (pp. 166–186). IGI Global.
Epstein, D., Pinho, I.D., Acosta, O.C., & Reategui, E. (2013). Inquiry-based learning environment using intelligent tutoring system. 2013 IEEE Frontiers in Education Conference (FIE), 1072–1074.
Fan, Z., Noller, Y., Dandekar, A., & Roychoudhury, A. (2023). Intelligent Tutoring System: Experience of Linking Software Engineering and Programming Teaching. ArXiv, abs/2310.05472.
Feng, S., Magana, A.J., & Kao, D. (2021). A Systematic Review of Literature on the Effectiveness of Intelligent Tutoring Systems in STEM. 2021 IEEE Frontiers in Education Conference (FIE), 1–9.
Francisco, R., & Silva, F. (2022). Intelligent Tutoring System for Computer Science Education and the Use of Artificial Intelligence: A Literature Review. International Conference on Computer Supported Education.
GAVRILOVIĆ, N., & JOVANOVIĆ, S. (2015, September). Implementing ITS (Intelligent Tutoring Systems) for Java programming e-learning. In The Sixth International Conference on e-Learning (eLearning-2015) (pp. 24–25).
Guang, G. (2003). Research on Intelligence-based Tutoring System. Journal of Chinese information processing.
Hieu, B.T. (2019). Analysis Of The Interoperability Of Intelligent Tutoring Systems For Programming: An Aspect Of Programming Exercises. International Journal of Scientific & Technology Research, 8, 655–659.
Hoppe, L., Gembarski, P.C., & Lachmayer, R. (2021). INTELLIGENT TUTORING SYSTEM AS A TOOL OF FORMATIVE ASSESSMENT IN DESIGN EDUCATION. DS 110: Proceedings of the 23rd International Conference on Engineering and Product Design Education (EPDE 2021).
Khazanchi, R., & Khazanchi, P. (2021). Artificial intelligence in education: A closer look into intelligent tutoring systems. In Handbook of research on critical issues in special education for school rehabilitation practices (pp. 256–277). IGI Global.
Lamia, M., Mohamed, H., & Samira, A. (2018, July). An Intelligent Tutoring System within a Social Network (IT2SN) for logical problem solvin. In Proceedings of the 5th Multidisciplinary International Social Networks Conference (pp. 1–6).
Lee, C., & Baba, M. S. (2005). The intelligent web-based tutoring system using the C + + standard template library. Malaysian Online Journal of Instructional Technology, 2(3), 34–42.
Moraes, L.O., & Pedreira, C.E. (2020). Designing an Intelligent Tutoring System Across Multiple Classes. CSEDM@EDM.
Müller, S., Bergande, B., & Brune, P. (2018, June). Robot tutoring: On the feasibility of using cognitive systems as tutors in introductory programming education: A teaching experiment. In Proceedings of the 3rd European Conference of Software Engineering Education (pp. 45–49).
Novickis, L., & Rikure, T. (2005). Intelligent Tutoring Systems. In Proceedings of IST4Balt International Workshop “IST 6th Framework Programme–Great Opportunity for Cooperation & Collaboration (pp. 35–40).
Orhun, E. (1991). Knowledge representation in intelligent tutoring systems for computer programming. American Journal of Mathematical and Management Sciences, 9, 243–259.
Rathore, A. S., & Arjaria, S. K. (2020). Intelligent tutoring system. In Utilizing educational data mining techniques for improved learning: emerging research and opportunities (pp. 121–144). IGI global.
Renumol, V. G., Jayaprakash, S., & Janakiram, D. (2009). Classification of Cognitive Difficulties of Students to Learn Computer Programming, Department of Computer Science, Indian Institute of Technology, Madras. India, Technical Report.
Rokade, P.S., & Jawandhiya, D. (2019). ONLINE INTELLIGENT TUTORING SYSTEM FOR IMPROVISING THE PEFORMANCE OF STUDENTS. Journal of emerging technologies and innovative research.
Šarić-Grgić, I., Grubišić, A., Stankov, S., & Stula, M. (2019). An agent-based intelligent tutoring systems review. Int. J. Learn. Technol., 14, 125–140.
Schez-Sobrino, S., Gmez-Portes, C., Vallejo, D., Glez-Morcillo, C., & Redondo, M. A. (2020). An intelligent tutoring system to facilitate the learning of programming through the usage of dynamic graphic visualizations. Applied sciences, 10(4), 1518.
Smith, P.J., Soloway, E., & Carroll, J.M. (1987). Special Session: Intelligent Tutoring and Help Systems. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 31, 280–280.
Sokolnicki, T. (1991). Towards knowledge-based tutors: a survey and appraisal of Intelligent Tutoring Systems. The Knowledge Engineering Review, 6, 59–95.
Swanson, T. (2019). The effectiveness of using intelligent tutoring systems to increase student achievement.
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and instruction, 4(4), 295–312.
Sweller, J. (2006). Discussion of ‘emerging topics in cognitive load research: using learner and information characteristics in the design of powerful learning environments’. Applied Cognitive Psychology, 20, 353–357.
William, F. K. A. (2024). Crafting a Strong Research Design: A Step-by-Step Journey in Academic Writing. International Journal of Scientific Research and Management, 12(3), 3238–3245.
William, F. K. A. (2024). My Data Are Ready, How Do I Analyze Them: Navigating Data Analysis in Social Science Research. International Journal of Scientific Research and Management, 12(3), 1730–1741.
Xhakaj, F., & Aleven, V. (2018). Towards improving introductory computer programming with an ITS for conceptual learning. In Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27–30, 2018, Proceedings, Part II 19 (pp. 535–538). Springer International Publishing.
Yousoof, M., & Sapiyan, M. (2016). CUSTOMIZED INSTRUCTIONAL PEDAGOGY IN LEARNING PROGRAMMING-PROPOSED MODEL. Journal of Theoretical and Applied Information Technology, 85(3), 309.
Yousoof, M., Sapiyan, M., & Kamaluddin, K. (2007). Measuring Cognitive Load- A solution to ease Learning of Programming. World Academy of Science, Engineering and Technology, International Journal of Social, Behavioral, Educational, Economic, Business and Industrial Engineering, 1, 32–35.
Yousoof, M., Sapiyan, M., & Oman, K. (2014, April). Cognitive load measurement in learning programming using NASA TLX rating scale (Non Physiological Measures). In Proceedings of the 13th International Conference on Applied Computer and Applied Computational Science, Kuala Lumpur, Malaysia (pp. 235–245).
Zhang, L., Zeng, Q., & Chen, Y. (2010). Research on the intelligent tutoring system based on teaching activities tree. 2010 International Conference on Computer Application and System Modeling (ICCASM 2010), 14, V14-534-V14-538.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
24 Sep, 2024
Reviews received at journal
03 Sep, 2024
Reviews received at journal
17 Aug, 2024
Reviewers agreed at journal
13 Aug, 2024
Reviewers agreed at journal
07 Aug, 2024
Reviewers invited by journal
31 Jul, 2024
Editor assigned by journal
31 Jul, 2024
Submission checks completed at journal
30 Jul, 2024
First submitted to journal
23 Jul, 2024

You are reading this latest preprint version

The Effectiveness of Intelligent Tutoring Systems in Supporting Students with Varying Levels of Programming Experience

Status:

Version 1

Abstract

Figures

1. Introduction

2. Literature review

2.1. Overview of Intelligent Tutoring Systems in Education

2.2. Discussing previous research on using ITS in programming education

3. Theoretical framework: Cognitive Load Theory

4. Methodology and data

5. Findings and discussions

5.1. Quantitative analysis

5.2. Qualitative analysis: Thematic analysis

5.3. Integration of the findings

5.4. Discussions

6. Conclusion

6.1. Theoretical contributions

6.2. Practical implications

6.3. Limitations and Prospective Research Opportunities

Declarations

References

Additional Declarations

Status:

Version 1