COVID-19 pandemics produced changes in social interactions and social distancing intended on reducing these interactions amongst people, causing education institutions to adopt the emergency remote teaching format. In this national higher education context, a survey carried out by the College of Deans of Undergraduate Studies (COGRAD-2020), linked to the National Association of Directors in Federal Institutions of Higher Education (ANDIFES), demonstrated that most of the IFES (Federal Institutions of Higher Education), planned complementary academic calendars that were completely remote or hybrid (Aguilera-Hermida, 2020; Ali, 2020; Buttler et al., 2021a; Chiou, 2020; Sarah I Hofer et al., 2021; Secundo et al., 2021a).
Due to the long duration of the suspension of in-person activities, the proper fulfillment of the academic calendar for the years 2020.1 and 2020.2 was disrupted. This situation became even worse because of the uncertainty regarding the resumption of academic in-person activities on the IFES. In addition, on April 28th 2020, CNE/MEC (National Council of Education) approved the opinion Nº 05, which covers the reorganization of the school calendar and the possibility of validation of non-face-to-face activities in order to meet the minimum annual class load.
Another aspect that required readjusting was the evaluation process, since it was conducted contemplating classroom teaching characteristics, the search for more precise indicators became necessary on the remote teaching assessment.
In the COVID-19 context, loss of teaching quality is projected due to the conditions for carrying out a semester entirely online in 2020. The present study aimed at devising and developing a measurement instrument to quantify students’ perception regarding the adaptation to the remote education provided by the Federal University of Amazonas – UFRA, over classroom-based learning.
Psychometrics can be understood as a theory or technique to assess psychological processes, usually applied in the areas of Psychology and Education (Pasquali, 2009). In this paper, the construct to be assessed is the perception of the students which represents a latent trait, such as its objectives, we have the quality evaluation of emergency remote teaching in Brazilian universities and the explanation of the meaning that individuals bestow to the answers on a variety of tasks through items. Thus, we expect psychometrics to assist the comprehension of mental processes through the quantitative method, which is an important aspect in science when seeking to depict the understanding of nature with greater precision.
The methodology applied was based on the creation of new instruments revised by (Coluci et al., 2015) and on mental health assessment principles (Gorenstein & Yuan-Pang, 2016), which present the theoretical procedures of design and construction of the psychological measurement proposed in (Pasquali, 2010). It is based on three poles: theoretical procedures, empirical procedures (experimental) and analytical procedures (statistical).
It is worth noticing that the multi-criteria proposed by (Nickel et al., 2010) was applied as complementary basis to the construction of the measuring instrument. Hence, it is necessary the identification of the requirements/needs of the students, definition of the requirements to compose the instrument and the definition of the goal-specifications that relate to the validation of the instrument by specialists. Finally, we sought to create an assessment instrument for the emergency remote teaching to validate the quality of this teaching and also, subsidize the various assessments and educational examinations in a large scale, such as: Basic Education System (SAEB), National Exam of Students Performance (ENADE), External on Site Evaluation of Higher Education Institutions, National High School Exam (ENEM) and Programme for International Student Assessment (PISA). Next, the bibliographic review is going to be demonstrated, the methodological steps to the construction and validation of the perception instrument about the quality of remote learning.
Literature review
Some studies about remote teaching were performed, among them the practice of psychological assessments, reinforcing the importance and need of researches that demonstrate accuracy, ethics and safety (Marasca et al., 2020), the investigation of students’ perception about the emergency online teaching (adoption, use and acceptance) and its quality (Aguilera-Hrmida, 2020; Sarah I. Hofer et al., 2021; Iglesias-Pradas et al., 2021; Moser et al., 2021; Secundo et al., 2021b; Shim & Lee, 2020) . Other articles indicate the psychological impacts experienced by higher education studets in the new reality which is a resultant of the fear of COVID-19 (Buttler et al., 2021b; Cavalheiro & Sticca, 2020; Faro et al., 2020; Shim & Lee, 2020).
There were few publications that analyzed the emergency remote teaching in Brazilian Universities (De la Fuente et al., 2021; Quispe-Pietro et al., 2021; Ribeiro & Corrêa, 2021), this can be based on two assumptions: 1) It is the first time the universities are experiencing this type of drasctic change on teaching setup, called Emergency Remote Teaching (ERT), which differs from the Distance Education, ofering a temporarty instructional support and developted with a reduced planning time (Hodges et al., 2020; Mohmmed et al., 2020; Whittle et al., 2020). 2) It is observed that the research focused on the impact of ERT, and directed to the Education literature, has not used psychometric procedures with an approach focused on latent trait theory and without validation performed with the support of mental health measurement fundamentals.
Moreover, papers or research measuring the quality of the adjustment to the remote teaching in the country were not identified, neither content where the methodology that had aspects related to psychometrics, method applied only in other countries using structural equation modeling (Buttler et al., 2021b). Therefore, the studies cited provided a bibliometric longitudinal framework, with themes related to: psychometrics, measurement, construction of measurement instruments and their validation, latent trait, quality of remote teaching and COVID-19, which finally supported and guided the construction of a measurement instrument with validation for this research.
Present study
Pratical motivating situation
The concern and relevance relative to the exceptional moment the world is experiencing in the face of COVID-19 pandemics reinforces the finding that it is necessary for universities to create alternatives in all the sectors they operate.
Thus, after the suspension of the Academic Calendar (AC) 2020.1 on March 19, 2020 and aiming on finding alternatives to face the situation, the Federal University of Amazonas – UFRA, located in the Northern region of Brazil, with its head office in Belém/PA, and five other campuses located in the cities of Parauapebas/PA, Paragominas/PA, Tomé-Açu/PA, Capitão Poço/PA and Capanema/PA, started the planning process for teaching.
Through Technical Note No. 01/2020 - PROEN/UFRA, dated March 26, 2020, the institution encouraged teachers to perform academic activities through the SIGAA platform, in order to maintain virtual contact with their students, such as the recommendation of topics for reading and activities.
Considering Resolution No. 567-Consepe/2020, of July 1, 2020, the UFRA, through Ordinance No. 943 of July 6, 2020, created a special committee. This committee proposed an alternative academic calendar, called Supplementary Academic Period (SAP) offering curricular components in emergency and temporary digital format, most importantly, seeking to keep the university active, notably from an undergraduate teaching perspective, where students and professors would maintain social, cognitive, and emotional interaction.
The Supplementary School Period (PLS - 2020.5), consisted of a reduced period of 8.5 weeks, which intended on minimizing the impacts of the long duration of the suspension of academic activities due to the COVID-19 pandemic, enabling the provision of curricular components and other academic activities in non-face-to-face format, synchronously or asynchronously, employing information and communication technologies (ICT). It is emphasized that PLS – 2020.5 did not reduce the load of curricular components but relaxed the shift and the offer period. The evaluation procedures are part of a branch that aims on evaluating the knowledge acquired by the students throughout their education. In contrast, subjective aspects and several factors directly impact the student’s leaning beyond the course’s own knowledge, particularly when concerning higher education.
Over the years, some factors have been investigated, such as reasons relating to evasion, social and economic impacts, home study environments, amongst others associated to the student’s experience in the academic environment (Ambiel, 2015; Buttler et al., 2021b; Newton, 2017; Sindhu et al., 2019).
Latent trait measured and assessment design
This study analyzes the students’ perspective and constitutes a property for evaluating the quality of education. Specifically, the object or psychological system to be represented is the latent trait “perception of students regarding the adequacy to remote teaching offered by the Federal University of Amazonas (UFRA), over the face-to-face teaching adopted before the COVID-19 pandemics context”. The properties or attributes of an object are the several aspects that characterize it and must be defined to the conception of an instrument.
Thus, given the latent trait, this can be measured by the adequacy attribute, and it is associated to a negative perception, neutral or positive, considering the following aspects of social and academic systems regarding: class dynamics, its own adjustment to remote teaching, structure and regulations of remote teaching, its own resources, study environment and external incentives, and finally, their own effort, relationships and personal motivations. As an example, the following model is presented in Figure 1:
This psychological attribute can be understood as a meter of latent trait of the evaluation of quality and adequacy of the remote teaching offered by the UFRA, which, in turn, it is not something directly observable or identifiable. Hence, it arises the need for the elaboration of a construct that manifests the reality observed indirectly, through other variables or factors that can be observed and are related to the subject of interest to be measured (Pasquali, 1997).
The adequacy concept or perception of adequacy, in the paper, refers to the idea of a search for criteria that measure the similarity between its knowledge and a corresponding object. Consequently, considering the remote teaching environment, in which the student is inserted and incorporating the academic and social integration aspects of the longitudinal model of (Tinto, 1975), the hypothesis of the conceptual model proposed (Figure 1) is that there is a set of factors that exert direct influence in this environment, and that, in turn, generates a negative perception, neutral or positive perception associated with the quality of remote teaching. Thus, the evaluation of the student, given the impact of these factors, will produce a worse adequacy perception, equivalent or better than face-to-face teaching, aiming at measuring the adequacy quality of the emergency remote teaching.
Based on the fact that the students’ perceptions about the educational environment influence directly their learning, (Roff et al., 1997) define educational “environment” or “climate” as any learning experience that influences the motivation of students to learn, affecting their attitudes, values and behaviors related to a learning task. Hence, measuring teaching quality from the students’ perspective corroborates as justification and contribution of this paper to the literature.
It is known from (Burleson & Thoron, 2014), that the premises of a good educational environment must include base-factors that are able to meet the following aspects: 1) the physical needs of students and associates regarding ergonomic conditions; 2) students’ safety and trust regarding the handling of the teaching/learning activities; 3) the engagement, the feeling of belonging in their development process as humans and the importance of performing professional activities to promote social, political, economic, cultural and environmental changes during the school term; 4) the self-esteem, linked to the importance of providing constructive feedbacks, given the context of the COVID-19 pandemics.
Multiple perspectives
Within this framework, and prior to the determination of the facts that compose this construct, the main motivation of this evaluation and measurement comes from the reports of students and teachers regarding the challenges of adjustment to the remote teaching methods applied. This exemplifies the importance of considering multiple perspectives on the evaluation purposes, thus, aiming to explicitly distinguish the perspectives, and in order to help the design of the evaluation (Newton, 2017), improve the way of designing the performance evaluation on the premises of jointly assess the perspectives of the concerned parties. Hence, it is possible to classify theses perspectives in three different types, which in this paper are presented as: teaching perspective, adequacy perspective and engagement perspective.
This way, since its pure intent idealizes the principle that the design of the measurement instrument must be guided by a single measurement purpose, that is, solely and exclusively to measure the latent trait, we have in this unity a test design that begins with the consideration of the expected interpretations to the intended uses of the scores obtained by the tests. The content and form of the test are then specified to provide evidence and support interpretations for the intended uses. When the scores of a single test are interpreted differently, to make different types of decisions, it is admitted that each intended interpretation must be independently validated (Association, 2018). In other words, a single perspective of the evaluation purposes is informed, and from a single point of perspective, their results are applied for different purposes.
The concept of multiple perspectives about the objectives of evaluation guides the test design to the intended uses of the scores, however it does not explicitly discuss how decision-oriented and content-oriented approaches intersect (Newton, 2017). The premise is that the instrument needs to provide information from multiple perspectives, in other words, the students have different perspectives, hence, they are capable of generating information that can be used by them, the teachers and the institution. The flowchart presented in Figure 2 exemplifies the relationship between multiple perspectives and factors associated with the latent trait.
The first perspective focuses on teaching and using the evaluation results in order to support, for example, the regulatory committee with reliable information, backed by quantitative methods and validated by evidences that aim on supporting the interpretation of the scores obtained from the tests. The designer is focused on the experience that must be attained by the students regarding their academic integration.
The second perspective focuses on the adequacy of the structure, regulations and technological resources of the UFRA, which is then, perceived by the students. The premise is that several students needed to adjust to the remote format, and by implication, derives from an institutional integration and its determinations. The designer is focused on the experience acquired by the students while adjusting to remote teaching, and from this perspective, the problem of how to specify the evaluation construct arises from the question of how to determine specific answers, that is, the instrument needs to minimize disturbances from other answers, and, most importantly, not to confuse aspects of the teaching perspective.
Finally, the third perspective of engagement focuses on the motivational consequences of the adequacy to remote teaching. The engagement operates mostly through social integration and the student’s agreement on being evaluated in terms of this new teaching structure. Therefore, it is assumed that the current situation of the COVID-19 pandemics tends to motivate students to study harder, or at least raise the comprehension of the need in keeping education active, that otherwise would have dedicated themselves to achieving learning results.
The designer is focused on capturing the engagement of the students and, given social and career aspects, the instrument needs to provide insights and some type of self-assessment of the student. So, the construct refers to the search of a set of answers that relates to the context of the pandemics, that is, the student will respond not as a direct comparison between teaching formats but context weighted.
According (Newton, 2017) researches that focus on performance evaluations and question the persistency of the purism of purpose need the coordination of the multiple perspectives of evaluation. In this sense, the construct can be defined and thus, determined as theoretial bases of the factors that compose it.
Finally, it is necessary to conceptualize in detail the construct, which will be done under two products: constitutive definitions and operational definitions. On the constitutive definition, it is understood that all the theoretical foundation, since the definition of the psychological object and the base-factors along with the three presented perspectives, provided subsidies for the creation of a conceptual model and aims at theoretical and abstract understanding, as well as the semantic validation of the instrument.
From an operational point of view, many instruments are available to assesses the educational environment, such as, in the field of medical sciences and related fields it is commonly evaluated applying DREEM (Roff et al., 1997) which reflects the overall construction of the teaching environment. Hence, given the specificity of the context and a target audience originated from a single higher education institution, this paper presents a procedure for the operationalization of the measurement instrument.