Learning Loss During COVID-19: An Early Systematic Review

doi:10.21203/rs.3.rs-518655/v1

With COVID-19 having caused significant disruption to the global education system, researchers are beginning to become concerned with the impact that this has had on student learning progress and, in particular, if learning loss has been experienced. To evaluate this, we conduct a thorough analysis of recorded learning loss evidence documented between March 2020 and March 2021. This systematic review aims to consolidate available data and document what has currently been reported in the literature. Given the novelty of the subject, eight studies were identified; seven of these found evidence of student learning loss amongst at least some of the participants, while one of the seven also found instances of learning gains in a particular subgroup. The remaining study found increased learning gains in their participants. Additionally, four of the studies observed increases in inequality where certain demographics of students experienced learning losses more significant than others. It is determined that further research is needed to increase the quantity of studies produced, their geographical focus, and the numbers of students they observe.

Educational Philosophy and Theory

Other Economics

Education

Learning loss

COVID-19

Systematic review

Due to the emergency nature of the COVID-19 pandemic, education systems around the world are facing extreme disruption. At its peak, UNESCO (2020) reported that nearly 1.6 billion learners in more than 190 countries, or 94 percent of the world’s student population, were impacted by educational institution closures. Given the abruptness of the situation, teachers and administrations were unprepared for this transition and were forced to build emergency remote learning systems almost immediately. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning progress or lack thereof.

The term “learning loss” is commonly used in the literature to describe declines in student knowledge and skills (Pier et al. 2021). Historic data provides researchers with information regarding where student learning should be year over year and is often measured through regular testing. Learning loss occurs when educational progress does not occur at the same rate it historically has compared to previous years (Pier et al. 2021).

Outside of the classroom, these losses may translate to greater long-term challenges. Currie and Thomas (2001) put this into perspective as they observe that a 0.20 decrease in standardized test scores could decrease future employment probability by 0.86 percent. Additionally, Chetty et al. (2014) observe that increasing student achievement by 0.20 standard deviations results on average, a 2.6 percent increase in annual lifetime earnings (Maldonado and De Witte 2020). Likewise, on average, another year of schooling is associated with an 8-9 percent gain in future earnings (Psacharopoulos and Patrinos 2018).

While many researchers have established learning loss prediction models (e.g., Azevedo et al. 2020), formal research and documentation of the actual impact that COVID-19 has had on student learning progress is just beginning to emerge. As the global education system continues to face pandemic related disruption, a strong understanding of how COVID-19 school closures are impacting student learning progress can better equip educators, policy makers, and researchers going forward.

Our focus in this paper is on the important research question: have COVD-19 education closures resulted in recorded student learning losses? To answer this, we conduct a thorough analysis of recorded learning loss evidence documented between March 1, 2020 and March 18, 2021. This systematic review aims to consolidate such data and document what has currently been reported in the literature. To the best of our knowledge, this is the first systematic review of its kind.

Our paper makes three important contributions. First, we develop a comprehensive review that consolidates the research that has been presented related to the impact of COVID-19 on student learning progress. Second, we answer the research question: has pandemic related learning loss been recorded in the literature? Third, based on our review, we identify significant gaps in the literature and provide relevant guidance for further research.

We begin by describing the methods used in this review to identify and collect the articles analyzed. We then present an analytical review where each article was categorized by its geographical region studied, length of school closure, education level of students analyzed, subject analyzed, documented learning impact, presence of increased inequality, and sample size. Next, we move on to our discussion where we review the findings of the analytical review. We then end by discussing areas for future research and summarizing the main ideas of this paper.

The initial search was performed using English-language articles published between March 1, 2020 and March 18, 2021. To ensure a comprehensive, multidisciplinary search, included electronic databases were ECONLIT, Google Scholar, PubMed, Education Resources Information Center (ERIC), and Cochrane Library. To conduct the search, the key words “covid-19”, “coronavirus”, “2019-ncov,” “sars-cov-2,” or “cov-19,” were used in combination with “learning loss,” “learning slide,” “education gap,” or “achievement gap.” Along with this, some studies were identified by reaching out to colleagues and researchers.

When conducting the search analysis, thousands of articles were identified; however, the majority of these pertained to hypothesized or predicted learning loss. To narrow this down to studies with recorded results, article abstracts were then screened. Studies which conducted student analyses and reported impacts on learning progress (either positive, negative, or insignificant) as a result of COVID-19 school disruptions were included. After this screening process, eight articles remained (Table 1). Factors for rejecting studies from our review included the absence of a student analyses and/or recorded impacts on learning progress, analyses occurring before the onset of COVID-19, or hypothesized results.

Table 1: Studies Included

Title	Authors
Learning inequality during the COVID-19 pandemic	Engzell, Frey, Verhagen 2020
Influence of COVID-19 confinement on students’ performance in higher education	Gonzalez, Rubia, Hincz, Comas-Lopez, Subirats, Fort, Sacha 2020
The impact of COVID‑19 on student learning in New South Wales primary schools: an empirical study	Gore, Fray, Miller, Harris, Taggart 2021
Collaborative for student growth. Learning during COVID-19: Initial findings on students’ reading and math achievement and growth	Kuhfeld, Tarasawa, Johnson, Ruzek, Lewis 2020
The effect of school closures on standardised student test outcomes	Maldonado, De Witte, 2020
Learning during the COVID-19 pandemic: It is not who you teach, but how you teach	Orlov, McKee, Berry, Boyle, DiCiccio, Ransom, Reese-Jones, Stoye 2020
Educational gains of in-person vs. distance learning in primary and secondary schools: a natural experiment during the COVID-19 pandemic school closures in Switzerland	Tomasik, Helbling, Moser 2020
Did students learn less during the COVID-19 pandemic? Reading and math competencies before and after the first pandemic wave	Schult, Mahler, Fauth, Lindner 2021

Once articles were selected, they were then coded using the classifications in Table 2.

Table 2: Classifications Used to Analyze Studies

Classification Term	Description
Country	The residing nation of the study’s participants.
Closure Length	The number of days that the participants were out of in person traditional schooling prior to assessment.
Education Level	Education level of participants.
Subject	Course subject of participants.
Learning Loss	Documented level of learning loss experienced by participants. If gains were experienced, “Improved” was listed.
Equality Impact	Documented differences in the level of loss experienced by certain groups of students.
Sample Size	Sample size of students analyzed.

As indicated in Table 3, we find that seven out of the eight studies identified learning loss amongst at least some of the students analyzed. For example, Maldonado and De Witte (2020) found Grade 6 students in Belgium experienced losses of 0.19 SD in math and 0.29 SD in Dutch. Engzell et al. (2021) find that overall, Grade 4-7 students in the Netherlands have encountered an average 0.08 SD learning loss in math, spelling, and reading. Tomasik et al. (2020) found learning progress of primary school students in Switzerland during in-person learning to be more than twice as high compared to the progress made during the eight-week school closure. Orlov et al. (2020) determined that economics students at four USA universities were 0.19 SD behind. Kuhfeld et al. (2020) found that Grade 3-8 students in the USA scored 5-10 percentile points below historic levels in math. Gore et al. (2021) found Year 3 students studying math in low ICSEA schools (Index of Community Socio-Educational Advantage schools) to be two months behind the progress students made in 2019 in Australia. Lastly, Schult et al. (2021) find learning losses of 0.07 SD in reading comprehension, 0.09 in operations, and 0.03 in numbers for Grade 5 students in Germany. At the university level, in a single university, for 458 students in STEM faculties, learning outcomes actually improved (Gonzalez et al. 2020).

Table 3: Results of Literature Classification

Source	Country	Closure Length	Education Level	Subject	Learning Loss	Equality Impact	Sample Size
Maldonado & De Witte, 2020	Belgium	9 weeks	Primary, Grade 6	Math	0.19 SD	Yes	Not specified
			Primary, Grade 6	Dutch	0.29 SD	Yes	Not specified
			Primary, Grade 6	Social Science	Insignificant	Not specified	Not specified
Engzell et al.,2021	Netherlands	8 weeks	Primary (Age 8)	Math	0.063 SD	Not specified	92180 students
			Primary (Age 8)	Reading	0.05725 SD	Not specified	76397 students
			Primary (Age 8)	Spelling	0.09375 SD	Not specified	90403 students
			Primary (Age 9)	Math	0.07325 SD	Not specified	93417 students
			Primary (Age 9)	Reading	0.0975 SD	Not specified	79016 students
			Primary (Age 9)	Spelling	0.07075 SD	Not specified	91567 students
			Primary (Age 10)	Math	0.0935 SD	Not specified	93769 students
			Primary (Age 10)	Reading	0.08425 SD	Not specified	68412 students
			Primary (Age 10)	Spelling	0.0755 SD	Not specified	91315 students
			Primary (Age 11)	Math	0.05025 SD	Not specified	73263 students
			Primary (Age 11)	Reading	0.07425 SD	Not specified	48537 students
			Primary (Age 11)	Spelling	0.07575 SD	Not specified	69841 students
			Primary (Grade 4-7)	Math, Spelling, Reading	0.08 SD	Yes	350 000 students
Tomasik et al., 2020	Switzerland	8 weeks	Primary (Grade 3-6)	Math, German	2X	Not specified	13134 Students
Tomasik et al., 2020			Secondary (Grade 7-9)	Math, German	Insignificant	Not specified	15551 Students
Gonzalez, et al. 2020	Spain	10 weeks	Higher ed	Applied Computing, Metabolism, Design of Water Treatment Facilities	Improved	Not specified	458 Students
Orlov et al., 2020	USA	3.5 weeks	Higher ed	Economics	0.185 SD	No	4 Universities
Kuhfeld et al., 2020	USA	Not specified	Primary (Grade 3-8)	Math	5-10 percentile points	Inconclusive	4.4 million students
Kuhfeld et al., 2020			Primary (Grade 3-8)	Reading	Insignificant	Inconclusive	4.4 million students
Gore et al., 2021	Australia	8-10 weeks	Primary (Year 3, all schools)	Math	Insignificant	Yes	1427 students
			Primary (Year 3, low ICSEA schools)	Math	2 months less growth	Yes	334 students
			Primary (Year 3, mid ICSEA schools)	Math	Improved, 2 months additional growth	Yes	813 students
			Primary (Year 3)	Reading	Insignificant	No	1429 students
			Primary (Year 4)	Math	Insignificant	No	1498 students
			Primary (Year 4)	Reading	Insignificant	No	1515 students
			Primary (Year 3-4)	Math and Reading	Insignificant	No	3030 students
Schult et al., 2021	Germany	8.5 weeks	Primary (Grade 5)	Math	0.09 SD 0.03 SD	Yes Yes	80000 students 80000 students
				Reading	0.07 SD	Yes	80000 students

Learning loss is being experienced. As Table 3. indicates, the early findings of seven studies provide evidence of experienced learning losses amongst students. These observed losses are occurring across a range of subjects, grade levels, and geographical regions. This signals that although robust and empirical research on COVID-19 related student learning loss is limited, learning loss itself may not be.

Not all students are experiencing learning loss. While the majority of the literature analyzed indicates that students are experiencing some level of learning loss, there were also instances where this was not the case. For example, both Maldonado and De Witte (2020) as well as Kuhfeld et al. (2020) found learning losses in certain subjects but insignificant impacts in others. Likewise, while Tomasik et al. (2020) found primary students to be impacted, they found no impact on secondary students. This is consistent with the literature showing that students in the early grades may be more vulnerable than secondary students because of their inability to seek learning on their own, due to the differences in developmental and cognitive abilities. In their Australian study, Gore et al. (2021) found overall there to be no evidence of learning loss in Year 3 and 4 students in math and reading with the exceptions being Year 3 students in math in low ICSEA schools who experiences losses and while mid ICSEA students experienced small gains. Lastly, in the case of Gonzalez et al. (2020) who studied university students in Spain, it was determined that student learning progress actually improved rather than declined during the COVID-19 learning disruption period, but this was for university students in STEM subjects at one university.

Some students are experiencing more learning loss than others. Of the eight studies, four found instances of in inequality, while only one exclusively found demographics to have no impact on learning loss. Gore et al. (2021) found instances of increased inequality as well as instances of no change. The other studies did not specify in this area or in the case of Kuhfeld et al. (2020) found inconclusive and minor differences between ethnic/racial groups. In the four studies where increases in inequality were observed, certain demographics of students experienced losses more significant than others. Maldonado and De Witte (2020) observed inequality within schools rise by 17 percent for math and 20 percent for Dutch. Engzell et al. (2021) determined that losses were up to 60 percent larger amongst students from uneducated homes. Gore et al. (2021) found the only losses to be amongst students from low ICSEA schools where the lower the ICSEA level the lower the educational advantage attending students have due to their parents’ occupation and education, their geographical location, and the school’s proportion of indigenous students (Australian Curriculum, Assessment and Reporting Authority, 2016). Schult et al. (2021) found losses in math amongst grade 5 students to be more severe in low achieving students. In reading comprehension Schult et al. (2021) found more severe losses amongst middle to high achieving students.

More research is needed. In general, the literature representing the impact that COVID-19 has had on student learning progress is limited in the quantity of studies available, geographical regions analyzed, and number of participating students.

Given the novelty of the subject, it is understandable why education researchers are only just beginning to analyze the learning losses that students have experienced. However, a stronger understanding of how COVID-19 school disruptions have impacted student learning is still needed. To support this, more studies are needed.

Along with this, the current studies that are available are limited in their geographical span. The only limited information that is currently available is from Belgium, the Netherlands, Switzerland, Spain, the United States, and Australia. Given the differences in educational institutions between countries, in terms of quality, length of school closures, and remote learning strategies, it is crucial that researchers continue to investigate COVID-19 related learning loss in countries where limited research exists.

Lastly many of the studies themselves that were analyzed in this systematic review had limited numbers of participants. For example, Gonzalez et al. (2020) analyzed just 458 students at 1 university. Similarly, Orlov et al. (2020) observed economics students in just 7 classes across 4 universities. While the information these studies presented remains relevant to their observed samples, research that can more accurately represent larger groups of students remains crucial to policy makers. As such, there is a demand for studies that analyze representative groups of students.

Through conducting a thorough analysis of recorded learning loss evidence documented between March 2020 and March 2021, this systematic review provided a consolidated audit of available research on COVID-19 related learning loss. Given the novelty of the subject, eight studies were identified; seven of the eight found evidence of student learning loss amongst participants, while one of these found instances of learning gains in a particular subgroup. The remaining study observed learning gains amongst university students. Along with this, four of the studies observed increases in inequality where certain demographics of students experienced learning losses more significant than others. Further research is needed to increase the quantity of studies produced, their geographical focus, and the numbers of students they observe.

Australian Curriculum, Assessment and Reporting Authority. (2015). What does the ICSEA

value mean? https://docs.acara.edu.au/resources/About_icsea_2014.pdf

Azevedo, J.P., Hasan, A., Goldemberg, D., Iqbal, S.A., & Geven, K. (2020). Simulating the

potential impacts of COVID-19 school closures on schooling and learning outcomes. World Bank Policy Research Working Paper No. 9284. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/33945

Currie, J., & Thomas, D. (2001). Early test scores, school quality and SES: Longrun

effects on wage and employment outcomes. Research in Labor Economics, 20, 103–132.

https://doi.org/10.1016/S0147-9121(01)20039-9

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers

II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), 2633–2679. https://doi.org/10.1257/aer.104.9.2633

Engzell, P., Frey, A., & Verhagen, M. (2020). Learning inequality during the COVID-19

pandemic. https://doi.org/10.31235/osf.io/ve4z7

Gonzalez, T., de la Rubia, M.A., Hincz, K.P., Comas-Lopez, M., Subirats, L., Fort, S., & Sacha,

G.M. (2020). Influence of COVID-19 confinement on students’ performance in higher education. PloS One. https://doi.org/10.1371/journal.pone.0239490

Gore, J., Fray, L., Miller, D., Harris, J., & Taggart, W. (2021). The impact of COVID‑19

on student learning in New South Wales primary schools: an empirical study. The Australian Educational Researcher. https://doi.org/10.1007/s13384-021-00436-w

Kuhfeld, M., Tarasawa, B., Johnson, A., Ruzek, E., & Lewis, K. (2020). Collaborative for

student growth. Learning during COVID-19: Initial findings on students’ reading and math achievement and growth. NWEA Research. https://www.nwea.org/content/uploads/2020/11/Collaborative-brief-Learning-during-COVID-19.NOV2020.pdf

Maldonado, J., & De Witte, K. (2020). The effect of school closures on standardised student

test outcomes. KU Leuven – Faculty of Economics and Business.

https://feb.kuleuven.be/research/economics/ces/documents/DPS/2020/dps2017.pdf

Orlov, G., McKee, D., Berry, J., Boyle, A., DiCiccio, T., Ransom, T., Reese-Jones, A., & Stoye,

J. (2020). Learning during the COVID-19 pandemic: It is not who you teach, but how you teach. Working Paper No. 28022. National Bureau of Economic Research. https://doi.org/10.3386/w28022

Pier, L., Hough, H.J., Christian, M., Bookman, N., Wilkenfeld, B., & Miller, R. (2021, January

25). COVID-19 and the educational equity crisis. Evidence on learning loss from the CORE data collaborative. PACE. https://www.edpolicyinca.org/newsroom/covid-19-

and-educational-equity-crisis

Psacharopoulos, G., & Patrinos, H. (2018). Returns to investment in education: a decennial review

of the global literature. Policy Research Working Paper No. 8402. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/29672

Schult, J., Mahler, N., Fauth, B., & Lindner, M. (2021). Did students learn less during the

COVID-19 pandemic? Reading and math competencies before and after the first pandemic wave. https://doi.org/10.31234/osf.io/pqtgf

Tomasik, M., Helbling, L., & Moser, U. (2020). Educational gains of in-person vs. distance

learning in primary and secondary schools: a natural experiment during the COVID-19 pandemic school closures in Switzerland. International Journal of Psychology. https://doi.org/10.1002/ijop.12728

UNESCO. (2020, June 6). UN Secretary-General warns of education catastrophe, pointing to

UNESCO estimate of 24 million learners at risk of dropping out. https://en.unesco.org/news/secretary-general-warns-education-catastrophe-pointing-unesco-estimate-24-million-learners-0

There is NO Competing Interest.

Learning Loss During COVID-19: An Early Systematic Review

Status:

Version 1

Abstract

Introduction

Methodology

Analytical Review

Discussion

Conclusion

References

Additional Declarations

Status:

Version 1