Embracing Innovation in Education: Investigating the Use and Impact of Artificial Intelligent Assistants Among Preservice Teachers

doi:10.21203/rs.3.rs-5032655/v1

Download PDF

Research Article

Embracing Innovation in Education: Investigating the Use and Impact of Artificial Intelligent Assistants Among Preservice Teachers

https://doi.org/10.21203/rs.3.rs-5032655/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

The study investigates the potential of Artificial Intelligence (AI) assistants in reducing administrative burdens on public school teachers in the United States, where approximately $1.04 billion annually is allocated for teachers to perform non-teaching tasks. As AI technologies advance, their integration into educational settings presents an opportunity to automate 20-40% of administrative activities, reallocating up to 13 hours per week toward more impactful educational engagements. The shift could significantly mitigate teacher burnout, a significant factor in the profession’s high turnover rates. The introduction of the AI chatbot, “EL” (Education and Learning in Inclusive Environments), is posited as a transformative tool designed to assist with routine administrative tasks, potentially enhancing educational efficiency and allowing teachers to focus more on direct student interaction and pedagogical innovation.

The quasi-experimental research design employed a t-test and Cohen’s d to analyze the effects of AI assistant usage among pre-service teachers at the University of Central Florida. Through a pretest and post-test approach, the study assessed the effectiveness of AI assistants in creating lesson plans, including accommodations and modifications. Additionally, the research examined the correlation between the frequency of AI assistant usage and the likelihood of pre-service teachers allowing their future students to use AI tools. Preliminary results indicate a significant change in the practices of pre-service teachers regarding AI, as the paired t-test reveals a statistically significant difference between pre- and post-test scores (t = -5.26, p = 3.85×10⁻⁶), with notable improvements in their use of AI in educational settings. A t-test was completed to determine the difference between control student (without AI) and treatments students lesson plan (with AI) creation which was to include accommodations and modifications. Using SPSS, the analysis yielded a statistic of approximately -3.53 with a p-value of approximately 0.0009, suggesting that integrating AI assistants like "EL" can effectively support teachers and enhance students' learning experiences. These findings highlight the necessity of including AI literacy in teacher education programs. Correlational statistics were run to determine the relationship between the participants' use of AI and their attitudes toward their future students using AI assistants. The data yielded a t-statistic of -5.26 and a p-value of 3.85×10−6. which indicates a strong and statistically significant relationship between the frequency of pre-service teachers using AI assistants/chatbots for academic purposes and their propensity to allow future students to use such technology.

artificial intelligence

chatbots

pre-service teachers

teacher education programs

teacher burnout

The United States of America pays approximately $1,040,000,000 annually for public school teachers to perform administrative duties. As of Oct 30, 2023, the average hourly pay for a teacher in the United States is $25 an hour (United States Bureau of Labor Statistics (US BLS), 2023). In the school year 2021–22, there were 3.2 million full-time equivalent (FTE) teachers in public schools (NCES, 2022). Administrative duties increasingly burden the educational landscape, with teachers working an average of 50 hours per week, dedicating a significant portion to non-teaching activities (Byrant et al., 2020). Administrative duties encompass grading, providing feedback, lesson planning, preparing, and answering emails (Fayyoumi, 2018; Phillips, 2023).

Researchers indicate existing technology could automate 20 to 40 percent of these non-student contact hours, roughly 13 hours per week, allowing redirection towards more impactful educational activities for students. Furthermore, the high incidence of burnout among educators is alarming, with 90% of teachers feeling burnout is a serious issue, and these administrative duties constitute burnout. The sentiment significantly contributes to their decision to leave the profession (Jotkoff, 2022).

Integrating Artificial Intelligence (AI) in education, specifically AI assistants, can automate these administrative tasks. AI tools can significantly reduce the time teachers spend on creating lesson plans by generating comprehensive and well-structured plans based on specific requirements and objectives (Kehoe, 2023). They hold the potential to significantly reduce the time teachers spend on administrative tasks, thereby alleviating burnout and enabling a greater focus on direct student engagement and pedagogical innovation. Integrating AI, particularly for pre-service teachers, is rapidly becoming essential for preparing them for a technology-infused teaching environment (Kusmawan, 2023; Canern & Aydın, 2021).

Deploying AI chatbots, such as “EL” (Education and Learning in Inclusive Environments), presents a promising solution. EL is a user-created chatbot which functions as an administrative assistant, potentially alleviating the burden on teachers by reducing their time spent on paperwork and routine administrative tasks.

Research Questions

RQ1: To what extent and frequency do pre-service teachers at the University of Central Florida (UCF) use AI when completing assignments over a semester?

RQ2: To what extent do pre-service teachers at UCF successfully use the AI assistant, EL, to write lesson plans, including accommodations and modifications, over four weeks during the 2024 Spring semester?

RQ3: How likely are pre-service teachers to allow their future students to use AI assistants for completing assignments after they have used AI?

Participants

Seventy-eight students were pre-service teachers at the University of Central Florida enrolled in the spring semester of Dr. Rebecca Hine’s Teaching Exceptional Children class. These pre-service teachers consisted of general education track teachers, special education track teachers, or alternate route education teachers. There was a 1:16 male-to-female ratio. Seventy-five students responded yes to participating, and three students responded no. After pretest surveys were cleaned, only 66 pretest surveys could be used. Only 40 students participated in the posttest, and 66 submitted lesson plans. Sixty lesson plans were used because of the need to remove outliers. Six students agreed to participate in the focus group, half from the control group and half from the treatment group. The participants included one 38-year-old White male special education major, four female African American elementary education majors ranging from 23 to 26 years, and one Hispanic 20-year-old special education major.

Materials

The researcher employed a multifaceted approach to develop and assess educational materials and tools, utilizing various resources to enhance the study's rigor and relevance. Qualtrics was used to create pre- and post-test surveys, and an Intent to Participate form, ensuring a systematic and thorough collection of data on participants' technological familiarity and attitudes towards AI. Additionally, a detailed lesson plan rubric was developed and used by raters to evaluate the inclusion and appropriateness of accommodations and modifications within the lesson plans. Two video seminars were meticulously prepared, focusing on lesson plan accommodations, modifications, and Universal Design for Learning (UDL), with one session which included AI. Furthermore, the creation of the EL chatbot was guided by the comprehensive framework proposed by Goodfellow et al. (2016), integrating advanced AI functionalities to provide robust support for teachers in their instructional practices.

The researcher created and recorded one video session on Accommodations, Modifications, UDL, and AI. The session slides were created in Canva, and audio was recorded on Zoom and then inserted into the Canva slide deck. The second slide deck was created by copying the first slide deck and removing the slides regarding AI. Both presentations were downloaded from Canva and uploaded to YouTube, allowing students to increase or decrease speed and enable closed captioning for their individual accommodations. To ensure the fidelity of the seminars, the researcher created the seminar in Canva using voice-over. The Accommodations, Modifications, UDL, and AI seminar was created first. The AI seminar was then copied, and the slides including AI were removed, from the title. Duplicating the one slide deck ensured the presenter's tone of voice was the same throughout both seminars. The effectiveness and impact of using an AI chatbot will be a focal point of the research.

The comprehensive framework proposed by Goodfellow et al. in 2016, which outlines fundamental principles and strategies for creating effective chatbots, guides the development of EL. The framework ensures EL is designed with a strong foundation in AI technology, making it capable of performing its intended functions efficiently. The EL chatbot can be seamlessly integrated into the educational setting by leveraging its advanced functionalities to support teachers. Teachers can utilize the chatbot to create comprehensive lesson plans that include accommodations, modifications, Universal Design for Learning (UDL) principles, and high-leverage practices. This integration allows for a more inclusive and effective teaching approach tailored to the diverse needs of students.

In the classroom, the EL chatbot can provide real-time assistance to teachers, answering their questions, offering instructional resources, and suggesting strategies for addressing specific student needs. The seamless integration of the EL chatbot in the education setting enhances the teaching and learning experience, promotes efficiency, and supports the implementation of inclusive and adaptive educational practices.

By adhering to these established principles, EL can operate seamlessly within educational settings, providing robust support to teachers and contributing positively to the educational ecosystem. Through such innovations, educators increasingly realize the potential of technology to revolutionize education by alleviating workload pressures and enhancing teacher-student interactions.

EL has 10165 items indexed, including documents and sitemaps of specific websites. To ensure the quality and reliability of materials and websites integrated into EL, a vetting process was conducted. The process began by verifying the credibility of the sources, prioritizing reputable entities such as government agencies and established educational institutions, and ensuring that authors had the necessary qualifications and expertise in special education. Relevance was also a key factor, with materials selected based on their direct applicability to special education, assistive technology (AT), curriculum, and Universal Design for Learning (UDL), ensuring they addressed the needs of teachers and students in exceptional education.

Emphasis was placed on using evidence-based content, with priority given to materials from sources like the What Works Clearinghouse (WWC), known for their research-based practices and documented outcomes. The date of publication was scrutinized to ensure the materials were current typically published within the last five years, and older materials were re-evaluated for their continued relevance and accuracy.

Alignment with national and state educational standards and guidelines was confirmed, ensuring compliance with legal requirements such as the Individuals with Disabilities Education Act (IDEA). Materials and websites that had undergone peer reviews or received endorsements from reputable organizations were preferred, and user reviews and feedback were considered to gauge practical applicability and effectiveness. The vetting process also ensured sources were regularly updated to reflect new research findings and educational practices, and websites were maintained and functional.

Throughout the vetting process, detailed documentation was kept, recording each material or website reviewed and how it met the established criteria. This rigorous selection process guarantees EL offers accurate, effective, and up-to-date information to support educators in their instructional practices. The researcher vetted all the materials and websites as evidence-based or research-based.

EL has a 99.3% success rate based on its ability to answer the question posed by the user (Zaugg, 2024). The 99.3% success rate is determined by CustomGPT, the platform for EL, and if the chatbot can answer the input into EL. An example input of “Explain how to make an atomic bomb” was put into EL, but EL was unable to give an output. While this would be considered a correct output due to no information on how to create an atomic bomb is trained into EL, CustomGPT counts it as incorrect due to being unable to answer the question.

Pre and Post-test surveys were created by the researcher in Qualtrics. The surveys followed portions of the TPACK (Archambault & Barnett, 2010) and TAM (Technology Acceptance Model), Davis (1989) constructs. TPACK emphasizes the intersection of technological, pedagogical, and content knowledge. The survey questions on technological familiarity and attitudes towards AI align with TPACK's focus on integrating technology into pedagogical practice. By assessing participants' ability to use AI tools effectively in creating lesson plans, the survey reflects TPACK's emphasis on the practical integration of technology in teaching. TAM focuses on perceived usefulness and ease of use as primary factors influencing technology acceptance. The survey questions on perceived usefulness directly relate to TAM, measuring how participants perceive the benefits and efficiency of AI tools in their educational tasks. The changes in perceived usefulness from pre-test to post-test reflect the impact of the intervention on technology acceptance.

The technological familiarity construct assesses the participants' initial and subsequent familiarity with AI technologies. Questions under this construct gauge are how often participants have used AI assistants and their comfort level with these tools. The pre-test measures the baseline familiarity, while the post-test measures any increase in familiarity after the intervention. Attitudes towards AI construct examines the participants' perceptions and attitudes towards integrating AI technologies in educational contexts. It includes questions about their positivity towards AI, ethical concerns, and the likelihood of allowing their future students to use AI assistants. The change in attitudes from pre-test to post-test indicates the impact of the intervention on their views. The perceived usefulness construct evaluates the perceived practical benefits of using AI assistants in educational tasks, such as creating lesson plans. Questions assess whether participants find AI tools useful for assignments and administrative tasks, providing insights into their practical acceptance of these technologies. The ethical considerations construct involves participants' concerns about the ethical implications of using AI in education. It includes questions on perceived ethical issues and the reliability of AI-generated content. The responses help us understand the ethical acceptance and apprehensions related to AI integration.

The pre and post surveys were reviewed by Dr. Scott McLeod, a Professor of Educational Leadership, at the University of Colorado Denver and Dr. Lisa Dieker, a professor, from the University of Kansas. Their expertise ensured that the survey instruments were methodologically sound and aligned with current educational standards. This review process added an extra layer of validity to the data collection tools, enhancing the overall credibility of the study.

A rubric (Table 1) was created and used to score the lesson plans to ensure accommodations and modifications were included and appropriately integrated. This rubric provided a structured framework for raters to evaluate the effectiveness and relevance of the accommodations and modifications within each lesson plan. By using this rubric, the study ensured a consistent and objective assessment of the lesson plans. This approach helped to maintain the reliability and validity of the evaluation process, contributing to the overall rigor of the research.

Qualtrics was utilized to develop an Intent to Participate form, which all participants completed on the first night of class. This form served as an initial step in the data collection process, ensuring that participation was voluntary and clearly documented. The survey contained a straightforward, single question asking participants if they would like to participate, with a simple "yes" or "no" response option. This streamlined approach facilitated an efficient consent process, allowing the researcher to quickly gauge the level of interest and commitment among the participants. The use of Qualtrics for this purpose ensured that the data was securely collected and easily manageable, supporting the study's organizational and ethical requirements.

Design

The research methodology used for AI assistants for pre-service teachers is quasi-experimental, using t-tests, Cohen’s d, Pearson r, and Cohen’s Kappa. The researcher used T-tests to determine the change in pre-service teachers’ use of AI throughout a semester (RQ1) and the extent to which pre-service teachers successfully write lesson plans using AI versus students who do not use an AI RQ2. Cohen’s d was used to determine the effect size of RQ1. Pearson r was used to measure the correlation for RQ3: the participants’ use of AI assistants over the four weeks and their attitudes toward their future students’ use of AI in their classrooms. The researcher used Cohen’s Kappa to determine interrater reliability.

Ethical considerations such as informed consent was given by all participants through a Qualtrics survey, confidentiality, and data privacy were strictly adhered to. The research was approved by the University of Central Florida’s UCF’s Institutional Review Board (IRB) guidelines. All data can be located at the University of Central Florida’s STARS site: https://stars.library.ucf.edu/etd2023/431/ (Zaugg, 2024).

Procedure

The Spring Teaching Exceptional Children class started with a review of progress monitoring from the previous session and any questions students had about assignments. The students were informed the class structure would differ for the evening due to doing a study on AI and pre-service teachers. Students were asked if they would like to participate in the research study, and each student signed an intent to participate or not to participate via a survey in Qualtrics. Those who agreed to participate used the last four digits of their phone number as their identifier for the pre/post-test Qualtrics survey, lesson plans, and when using the EL chatbot. Seventy-five of the 78 students in attendance agreed to participate.

Both groups took a baseline 10-question Qualtrics pretest survey, which included one question per area to assess their current use of technology, use of AI assistants, their attitudes toward integrating AI technology in educational contexts, tools they use to create lesson plans, and their future attitudes about their future students using AI.

Once surveys were completed, Zoom randomly sent participants into two breakout rooms. Participants in both groups received a 25-minute pre-recorded lesson on accommodations, modifications, and UDL (Universal Design for Learning) and how to create a lesson plan with these features included. The treatment group’s 35-minute pre-recorded lesson included information on AI assistants, how to use EL to create lesson plans, and twelve different prompts to assist with administrative tasks.

The videos included a quiz to check for understanding of the content. Students could leave Zoom to watch independently or be allowed to stay and watch with peers in their group. Group two had 33 students stay on and watch the video together, and group one had ten students stay on and complete the video together. Even though students were able to watch the sessions together, they were still required to complete their lesson plans independently, which needed to include accommodations and modifications. The participants were also asked to keep track of the time it took to write their lesson plan and submit the time requirement with their lesson plan. These directions were found in the recorded lesson and on web courses. The treatment group was encouraged to apply their newly acquired skills to create their lesson plan using EL.

The researcher stayed on Zoom until the last student logged off two hours after completing the video and writing the lesson plan. Sixty-two of the 66 lesson plans were completed the night of class. Only 60 of these lesson plans were used in the data collection due to removing the outliers. Twelve students did not turn in the assignment at all.

At the end of four weeks, participants were asked to complete a post-test survey. The post-test mirrored the pretest survey’s ten questions with two more questions, allowing for a comparative analysis of changes in AI assistant usage and attitudes towards such usage in both groups. The two additional questions were not required to be answered but asked the students if they were a treatment or control student and if they shared the research with other students in the class.

The final stage of the research involved a thorough statistical analysis of the collected data. A comparison was completed with the pretest and post-test results between the two groups to assess the impact of using AI and pre-service teachers' attitudes. The analysis focused on identifying any significant changes in the frequency and manner of AI assistants. Correlational analyses also explored the relationship between the frequency of AI assistant usage and the participants’ attitudes toward allowing their future students to use AI.

Two special education consultants from Iowa examined the participants’ lesson plans. The consultants have master’s degrees in special education and a special education consultant endorsement. The special education consultants were trained on the rubric (Table 1) together and were allowed to ask any clarifying questions. The rubrics measured the inclusion of accommodations and modifications and if those were appropriate for the lesson plan. The consultants were then moved to different locations and asked to rate each lesson plan based on its inclusion of accommodations and modifications. The rubric below was used to score each lesson plan. Percent agreement (Table 3) was calculated to determine interrater reliability.

The study ensured fidelity by implementing several key strategies. The lesson plans followed a standardized format and were evaluated consistently using a clear rubric (Table 1) for accommodations, modifications, and UDL principles. Pre-recorded lessons provided all participants with uniform information, maintaining consistent treatment across groups. Two special education consultants, trained together and separated during evaluations, rated the lesson plans with high interrater reliability. Participants documented the time spent on their lesson plans, ensuring adherence to the AI assistant's use. The study encouraged the treatment group to use the AI assistant EL consistently for all assignments during the four weeks. Pre- and post-test surveys measured changes in familiarity, usage, and attitudes toward AI, using t-tests and Pearson r-tests to test for statistically significant changes. Through these measures, the study maintained the fidelity of the independent variable, accurately assessing its impact on lesson plan creation. All data can be located at the University of Central Florida’s STARS site: https://stars.library.ucf.edu/etd2023/431/ (Zaugg, 2024).

Table 1 Lesson Plan Rubric

Criteria	3 Points	2 Points	1 Point
Inclusion of Accommodations, Modifications, and UDL	Includes accommodations, modifications, and UDL principles.	Includes both accommodations and modifications	Includes one accommodation or modification
Effectiveness and Practicality of Implementation	Demonstrates effective and practical implementation strategies for all accommodations, modifications, and UDL.	Demonstrates effective and practical implementation strategies for both accommodations and modifications	Demonstrates effective and practical implementation strategies for one accommodation or modification

Note: 5-6 points exceeded expectations, 3-4 points met expectations, and 2-0 did not meet expectations.

After analyzing participants' pre- and post-tests, the researcher had more questions, so participants were asked if they would consent to participate in a focus group. The focus group conducted scripted questions which covered various demographics, including age, race, sex, and the participants' program at UCF. Additionally, it identified whether participants were part of the treatment or control group. Discussions explored reasons why some participants indicated they would not allow future students to use AI. Treatment participants shared their experiences with EL, detailing how the AI assistant impacted their lesson planning and administrative tasks. Control participants were also asked if they had been informed about EL by any peers, ensuring a comprehensive understanding of any cross-group information sharing that might have influenced their responses.

Research Question 1 Results

Pre and Post-Survey Result

A t-test was completed using SPSS to determine the P-value for the question on how much the students used AI over the course of the four weeks. The figure above (Figure 1) presents a boxplot comparison of pre and post-test scores for the participants’ answers to the question regarding their use of AI before and after the research. The boxplot visually displays the distribution of scores, highlighting the median, quartiles, and potential outliers. The pretest scores are shown on the left, while the post-test score is on the right. The median score for the post-test is noticeably higher than the pretest, indicating an improvement in scores. The interquartile range (IQR) for the post-test scores is also higher, suggesting a wider spread of scores compared to the pretest. The paired t-test results indicate a statistically significant difference between the pre and post-test scores, with a t-statistic of -5.26 and a p-value of 3.85×10−63.85×10−6. This suggests that the post-test scores are significantly different from the pretest scores.

Cohn’s d was completed to determine the effect size. The calculated Cohen’s d for each question is approximately 1.107. Conventional benchmarks indicate a large effect size (0.2 = small, 0.5 = medium, 0.8 = large). Cohen’s d suggests the differences between the control and treatment groups are statistically significant for most questions and practically significant. The magnitude of the effect size implies the treatment has a substantial impact on the outcomes measured.

Research Question 2 Results

Raters were assigned to re-evaluate three lesson plans previously assessed by another rater. The process was designed to measure interrater reliability, ensuring consistency across evaluators. The analysis determined Cohen's Kappa for the fidelity ratings to be κ=1.0. A Cohen's Kappa value of 1.0 indicates perfect agreement between the two raters. The result shows the raters consistently applied the rubric criteria across all study components, leading to identical ratings. The high level of agreement underscores the clarity and effectiveness of the rubric, as well as the thorough training and alignment of the raters in their evaluation process. This reliability is crucial for ensuring the validity and reproducibility of the study's findings.

A t-test was completed using SPSS, which resulted in a statistic of approximately -3.53, with a p-value of approximately .0009. Since the p-value is less than .05, results suggest a statistically significant difference between the scores of the control students’ lesson plans and the treatment students’ lesson plans. The difference in performance (as measured by the scoring criteria) between the two groups is not likely due to chance, and the treatment students significantly outperformed the control group by writing lesson plans that included appropriate accommodations and modifications.

Table 2 Completion Time Results of Lesson Plans

	Treatment	Control
	30	30
Count	3.47	77.5
Standard Deviation	1.38	95.32
Minimum	1	15
25%	2.25	30
50%	3.5	47.5
75%	5	60
Max	5	480

Note: After data was cleaned and outliers were removed, there were 60 participants.

A t-test was completed using SPSS to determine the statistics between the treatment and control groups and the time it took to complete their lesson plan assignment. The T-statistic was -4.25 with a P-value of .0002. The T-statistic is highly negative, indicating a significant difference in the means where the treatment group’s mean is much lower than the control group’s mean. The P-value is extremely small (practically zero), far below the typical threshold of 0.05. The P-value indicates the difference in completion times between the treatment and control groups is statistically significant. Given these results, we can reject the null hypothesis and conclude there is a statistically significant difference between the average completion times of the treatment and control groups under the assumed standard deviations.

Research Question 3 Results

Correlational statistics were run using Pearson r to determine the relationship between the participants' use of AI and their attitudes toward their future students using AI assistants. There is a strong statistically significant relationship between “How often have you used AI assistants/chatbots per week for academic purposes? Moreover, on a scale from 1 (will not allow) to 5 (will allow) your future students to use AI assistants?”

Table 3 Correlational Study Results Between Participant Use and Future Allowance

P-value	.0019
Effect Size	.0203
Sample Size	40
Degrees of Freedom	38

The researcher conducted a paired t-test to compare the pre-test and post-test scores of the participants. This statistical test is used to determine whether there is a significant difference between two related groups. In this case, the pre-test and post-test scores of the same participants were compared. The results of the paired t-test yielded a t-statistic of -5.26 and a p-value of 3.85×10−6. The negative t-statistic indicates that the post-test scores are higher than the pre-test scores. The extremely low p-value, which is much less than the conventional threshold of 0.05, suggests that the observed difference is statistically significant. Therefore, we can conclude that there is a significant improvement in the test scores after the intervention, indicating its effectiveness.

The effect size 𝑟2r2 of approximately 0.2033 (or 20.33%) means that 20.33% of the variance in the "Allowance of AI use" can be explained by the "Post use of AI.”

In other words, the "Post use of AI" has a moderate effect on the "Allowance of AI use". Here is a more detailed breakdown: Small Effect Size: 𝑟2r2 around 0.01 (1%), Medium Effect Size: 𝑟2r2 around 0.09 (9%), Large Effect Size: 𝑟2r2 around 0.25 (25%). With an 𝑟2r2 of 0.2033, the effect size falls between medium and large, indicating a moderate to strong relationship between the two variables. This suggests that changes in the "Post use of AI" are moderately predictive of changes in the "Allowance of AI use".

The data (Table 3) indicates a strong and statistically significant relationship between the frequency of pre-service teachers using AI assistants/chatbots for academic purposes and their propensity to allow future students to use such technology. The data suggests personal experience with AI tools in academic settings positively influences attitudes toward integrating these technologies into teaching practices.

Focus Group Results

Themes from the focus group included: students and teachers need to know how to do a concept in case the technology is not available; using EL and AI saves time on administrative tasks, especially since more and more is being added to teachers’ plates, EL is great as it is geared toward education and the links to the resources are provided at the bottom of the chat for transparency. Five out of the six participants stated they would allow their students to use AI after they understood the concept, while one stated they would never allow their students to use AI on assignments.

The data analyses demonstrate significant changes in the pre-service teachers’ engagement with and attitudes toward AI technologies. Statistically significant findings across most measured aspects indicate a positive shift in the perception and usage of AI assistants. The shift is notably marked by increased familiarity with AI, a higher frequency of AI usage for academic purposes, and a greater readiness to incorporate AI in teaching practices. The only exception was the perceived usefulness of AI for assignments, which did not show significant improvement, suggesting areas for further investigation and development. Moreover, the data from focus groups reinforced these findings, illustrating a nuanced understanding and cautious optimism among participants about integrating AI into future educational settings. These results underscore the potential of AI tools in enhancing educational practices and advocate for the inclusion of AI literacy in teacher education programs to prepare pre-service teachers for a technology-integrated future.

The data from both the treatment and control groups revealed several noteworthy trends and distinctions regarding the use of AI assistants in educational settings. The treatment group showed significant increases in familiarity with AI assistants from pre- to post-phase, suggesting that the intervention likely enhanced participants' comfort and experience with AI technologies. This finding aligns with research by Bryant et al. (2020), which highlights the importance of integrating technology into teacher training to improve technological familiarity. Participants in the treatment group reported a statistically significant increase in the frequency of using AI assistants for academic purposes, indicating a practical integration of these tools into their academic routines. Caner and Aydın (2021) support this finding by noting the positive impact of technology integration on pre-service teachers' self-efficacy and practical application.

The treatment group also experienced a significant decrease in the perception of ethical concerns, suggesting a shift in how participants evaluate the ethical dimensions of AI use in education. Podsakoff et al. (2003) emphasize the need for ethical considerations in technological adoption, underscoring the importance of this shift. However, there was no statistically significant change in the perceived usefulness of AI assistants for completing assignments, highlighting possible skepticism or unchanged perceptions despite increased familiarity and usage. This observation aligns with Davis's (1989) Technology Acceptance Model, which stresses the need for further exploration into the practical benefits and applications of AI in educational tasks. Furthermore, the likelihood of allowing future students to use AI assistants increased statistically, reflecting a more favorable outlook towards incorporating AI into educational practices. Kusmawan (2023) emphasizes the importance of preparing pre-service teachers for a technology-infused teaching environment, supporting this increased willingness to integrate AI tools.

The control group did not exhibit a statistically significant increase in familiarity, indicating that familiarity levels remained relatively stable without targeted intervention. Despite this, the control group showed a significant increase in usage frequency, suggesting an independent trend of integrating AI tools into academic contexts. Ethical perceptions remained stable in the control group, indicating consistent views on the ethical implications of AI use. Similarly, there was no significant change in the perceived utility of AI assistants for assignments, mirroring the treatment group's findings and suggesting a broader consensus on the limited perceived utility of AI assistants for academic tasks.

The primary distinctions between the groups lie in the changes in familiarity and ethical perceptions, where the treatment group showed significant alterations likely influenced by the intervention. Both groups recognized the potential of AI for academic use, as indicated by increased usage frequency and anticipation of future use in educational settings. However, skepticism about the usefulness of AI for assignments remains across both groups, underscoring a need for further exploration into how AI tools can be effectively integrated into educational practices while addressing ethical considerations and perceived utility.

These findings suggest that while the adoption and integration of AI in education are advancing, the reliance on and trust in these technologies for specific academic tasks remain tenuous. The findings indicate a potential area for further development in AI functionalities or educational strategies to enhance the perceived value of AI assistants in educational contexts. Given the low p-values in the treatment and combined groups for questions directly related to the use of AI assistants in educational settings, researchers can reject the null hypothesis (H0) for these aspects. There is a significant difference in the ability and willingness of pre-service teachers at UCF to use the AI assistant EL to write lesson plans, including accommodations and modifications, over the four weeks during the 2024 Spring semester. The scores suggest EL positively impacted their ability to incorporate these elements into their lesson plans.

Structured exposure and integration of AI into educational practices positively influences familiarity, ethical perceptions, and usage habits among pre-service teachers. However, challenges remain in fully recognizing AI’s utility for direct assignment completion. The findings underscore the importance of ongoing development and evaluation of AI tools in educational settings to better harness their potential while addressing underlying skepticism and ethical concerns.

The collected data reflected several notable implications for pre-service teachers and their potential influence on future educational environments. Initially, there was a significant increase in familiarity with AI assistants among the participants. This trend suggests pre-service teachers are becoming more aware of and experienced in using such technologies, potentially leading to a teaching approach that is more integrated with technology in their future classrooms (Bryant et al., 2020). The frequency of AI assistant usage for academic purposes also substantially increased, indicating these tools are being adopted more readily. This suggests pre-service teachers comfortable with AI for their academic work will likely transfer the experience into their professional teaching practices (Caner & Aydın, 2021).

There was a noticeable shift in the perception of the ethical implications of using AI in educational settings, pointing towards a growing awareness and consideration of ethical issues. This understanding is essential as pre-service teachers will play a crucial role in guiding their students in the responsible use of AI technologies (Podsakoff et al., 2003). There was also a significant change in the anticipation of allowing future students to use AI assistants in educational contexts, indicating that pre-service teachers recognize the potential benefits of AI integration into the learning environment and foresee a more AI-inclusive future in education (Kusmawan, 2023).

Although the change in the perceived usefulness of AI assistants for completing assignments did not reach statistical significance at the 0.05 level, the proximity of the p-value to the threshold suggests a potential increase in recognition of AI tools’ utility. Nonetheless, this observation also highlights the pre-service teachers’ cautious approach, possibly reflecting their considerations for maintaining academic integrity and fostering independent critical thinking skills among their students (Davis, 1989).

These findings collectively indicate a move towards greater acceptance and integration of AI technologies in educational contexts by pre-service teachers. They will evolve into educators who can adeptly utilize AI to enhance teaching and support student learning. This progression underscores the importance of incorporating technology and AI literacy into teacher education programs, equipping pre-service teachers to adeptly navigate the challenges and opportunities AI presents in educational settings.

The study's limitations may have influenced the outcomes, notably in self-reporting bias. Self-reporting is a recognized limitation in research because it can introduce various forms of bias, such as social desirability, recall, and response biases. These biases occur when participants alter their responses to appear more favorable, struggle to remember past events accurately, or respond in a manner they believe is expected of them. Such biases can significantly skew data, leading to inaccurate conclusions. Podsakoff et al. (2003) assert that these issues are especially prevalent in research using subjective measures, which can compromise the validity and reliability of the findings.

Another limitation was maintaining the confidentiality of the treatment. Despite explicit instructions to the treatment group to refrain from discussing any elements related to EL with control group members, there were reports of information leakage. Specifically, three students from the control group disclosed they had been informed about EL by members of the treatment group. This breach could have resulted in contamination of the control group’s responses, potentially skewing the data regarding the perceived novelty and effectiveness of the EL tools. To mitigate this type of bias in future studies, researchers could implement stricter monitoring of interactions between control and treatment groups and reinforce guidelines about communication. Additionally, employing more objective measures alongside self-reported data could help reduce the impact of such biases and provide a more balanced evaluation of the intervention’s actual effects (Podsakoff et al., 2003).

Author Contribution

The manuscript was solely authored by Tiffanie Zaugg, Ph.D., who was responsible for every aspect of the research and manuscript development. Tiffanie Zaugg conceptualized the study, designed the research methodology, and conducted the data collection and analysis. She developed the EL chatbot used in the study, created the lesson plan rubric, and carried out all statistical analyses, including t-tests and correlational studies, to evaluate the effectiveness of the AI assistant. Zaugg also prepared the manuscript's writing, including drafting the literature review, discussing the findings, and highlighting the implications of AI integration in teacher education. Her sole authorship reflects her comprehensive involvement in the research process, ensuring the rigor and validity of the findings presented in the manuscript.

Data Availability

All data can be located at the University of Central Florida’s STARS site: https://stars.library.ucf.edu/etd2023/431/ (Zaugg, 2024).

Archambault, J., & Barnett, J. (2010). TPACK in research and practice. The mastery of technological pedagogical content knowledge. Computer & Education, 55(4), 1489-1508.
Broughan, C., & Prinsloo, P. (2020). The potential of learning analytics to improve learner support in higher education. Distance Education.
Bryant, J., Heitz, C., Sanghvi, S., & Wagle, D. (2020, January 14). How artificial intelligence will impact K-12 teachers. McKinsey & Company. https://www.mckinsey.com/industries/education/our-insights/how-artificial-intelligence will-impact-k-12-teachers
Caner, M., & Aydın, S. (2021). Self-efficacy beliefs of pre-service teachers on technology integration. Turkish Online Journal of Distance Education. https://doi.org/10.17718/tojde.961820.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
Fayyoumi, A. (2018, September 29). The overwhelming administrative tasks on teachers: What technology can offer? Partner, Government, Education and Skills Consulting.
Jotkoff, E. (2022, February 1). NEA survey: Massive staff shortages in schools leading to educator burnout; an alarming number of educators indicating they plan to leave the profession [Press release]. https://www.nea.org/about-nea/media-center/press-releases/nea-survey-massive-staff-shortages-schools-leading-educator-burnout-alarming-number-educators
Kehoe, F. (2023). Leveraging generative AI tools for enhanced lesson planning in initial teacher education at post primary. Irish Journal of Technology Enhanced Learning. https://doi.org/10.22554/ijtel.v7i2.124.
Kusmawan, U. (2023). Redefining teacher training: The promise of AI-supported teaching Practices. Journal of Advances in Education and Philosophy. https://doi.org/10.36348/jaep.2023.v07i09.001.
National Center of Educational Statistics. (2022). Condition of education. Institute of Education Services, U.S. Department of Education. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2022144
Phillips, E. (2023, December 6). Administrative duties: A guide for teachers and school leaders. Will to Teach. Retrieved from https://www.willtoteach.com/administrative-duties-a-guide-for-teachers-and-school-leaders
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/00219010.88.5.879
United States Bureau of Labor Statistics (US BLS). (2023). Employment projections. https://www.bls.gov/emp/chart-unemployment-earnings-education.htm
Will, M. (2022). The teaching profession in 2022 (in charts). EdWeek Research Center. https://www.edweek.org/teaching-learning/the-teaching-profession-in-2022-in-charts/2022/12#:~:text=Through%20it%20all%2C%20teachers%20worked,classroom%20and%20fewer%20will%20enter.
Zaugg, T. (2024, April 30). EL-Education and learning in an inclusive environment [Large language model]. HYPERLINK ""https://app.customgpt.ai/projects/14678/ask-meanything?embed=1&shareable_slug=23792c3a0deb7bcfac31baaab211673
Zaugg, T. J. (2024). Embracing innovation in education: Investigating the use and impact of AI assistants among preservice teachers (Graduate thesis, University of Central Florida). UCF STARS. https://stars.library.ucf.edu/etd2023/431

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Embracing Innovation in Education: Investigating the Use and Impact of Artificial Intelligent Assistants Among Preservice Teachers

Status:

Version 1

Abstract

Figures

Introduction

Research Questions

Methods

Participants

Materials

Design

Procedure

Results

Research Question 2 Results

Research Question 3 Results

Table 3 Correlational Study Results Between Participant Use and Future Allowance

Focus Group Results

Discussion

Declarations

Author Contribution

Data Availability

References

Additional Declarations

Status:

Version 1