a. Study Design
The purpose of this study was to examine students’ academic performance in VCAOL during the Covid-19 pandemic. Academic performance was chosen as the subject of study by researchers because many students were affected by their academic performance, or it took time to adjust to online learning during the Covid-19 outbreak [2]. From September 2022 to January 2023, data was collected from university students at numerous universities in the Jakarta area using Google Form questionnaire. The questionnaire has 28 statements filled up. The 28 statements examined students' perceptions of VCAOL during the Covid-19 pandemic, including 4 perspectives namely video conference application (VC), internet connection (IC), student ability to study (SL), learning method (LM), and student knowledge (SK). There were 361 data gathered as a dataset. Following that, 28 statements on questionnaire were regarded as features for developing machine learning prediction models. Because supervised prediction required attribute as label prediction, as a label prediction, one statement, students’ academic performance in VCAOL during Covid-19, was chosen, with options 'very degraded', 'decreased', 'stable', 'good', and 'very good'. This label represented a student's statement for academic performance during the Covid-19 pandemic. Features for prediction model are shown in Table 2. Subsequently the flowchart of the proposed work displays in Fig. 1.
Table 2 Features perspective
Perspective
|
Features
|
Video conference application (VC) [19]
|
Benefit
Efficient
ListApplicationVicon
Frequency
EaseOfLearning
EaseWorkGroup
|
UserInterface
Bored
Interactive
Project
Feature
Tools
|
Internet connection (IC) [20]
|
NetworkNot Affect
NetworkAffects Speed
|
Often Disconnected
Internet Problems
|
Sudents’ ability to learn (SL) [19]
|
ConstraintType
FrequencyConstraint
|
Completing Project
|
Learning method (LM) [19]
|
LearningAsUsual
AdequateMethod
|
SupportingMaterial
IncreaseValue
Ability
|
Student knowledge (SK) [20,21]
|
Performance
Knowledge
|
Competence
Positive effect
|
b. Data Pre-processing
The data pre-processing step includes data quality assessment, data cleansing, data transformation, and data reduction [22]. Because all the statements in the questionnaire were previously designated as required inquiries, all inquiries must be answered, as a result, the missing value was not found during this activity. Subsequently, each statement was distinct, no duplicate statements were found, and hence no duplicate values were found as well. We determined the data type of each feature during data pre-processing; we detected 2 features on ordinal data type, 20 features on numerical data type, and 6 features on categorical data type. Following that, we transformed features for data compatibility. We executed mandatory changes for data compatibility by converting non-numeric features to numeric ones. We executed the change prior to training. Table 3 shows the converting value for each feature.
Table 3 Data transformation
Data type
|
Data transformation
|
Data type
|
Ordinal
(n = 2)
|
Value:
Benefits attribute contains values: Very useful, moderately useful, Helpful, less useful.
Efficient attribute contains the value:
Very efficient, moderately efficient, Efficient, less efficient
Data transformation:
The value is converted to a number between 1 and 4, with 1 representing the lowest order level and 4 representing the highest order level.
|
Ordinal
(n = 2)
|
Numerical
(n = 1)
|
Value:
Frequency contains integers: 0, 1, 2, 3 and >3
Data Transformation:
Numbers 0, 1, 2, and 3 are unchanged, while number > 3 is changed to number 4.
|
Numerical
(n = 1)
|
Categorical
(n = 6)
|
Value:
Data in the form of options with categorical data types allows the user to select more than one option.
Data transformation:
Values are translated into numbers based on the number of choices selected in each statement.
|
Categorical
(n = 6)
|
Numerical
(n = 19)
|
Value:
The data are derived from the numbers on a Likert scale that ranges from 1 to 5.
Data transformation:
Keep the numbers entered in each statement unchanged.
|
Numerical
(n = 19)
|
As a label prediction, one statement, students’ students’ academic performance in VCAOL during Covid-19, was chosen, with options 'very degraded', 'decreased', 'stable', 'good', and 'very good'. We converted each option into number: 'very degraded': 0, 'decreased': 1, 'stable': 2, 'good': 3, 'very good': 4.
c. Data splitting
To train any ML model, irrespective of the nature of the dataset used, the dataset must be split into training and testing data. In a data split, the training data set is used to train and construct models. Training sets are frequently used to estimate different parameters or to compare different model performances [23]. We split the data using the hold-out method, which typically uses 80% of data for training and the remaining 20% of the data for testing.
d. Machine Learning Algorithms for Prediction
In the classification experiments, RF, SVM, and GNB were used. Based on machine learning, these algorithms have the potential to be used to predict students' academic performance [12-18]. RF is an ensemble of high-performing trees that have been combined into a single model. This approach outperforms the decision tree technique in terms of performance [24]. The support vector machine (SVM) is a class of supervised learning methods that can be used for classification, regression, and outlier detection. SVM techniques employ many kernel functions, the most prevalent of which are linear, nonlinear, polynomial, radial basis functions, and sigmoid functions. This kernel function is analogous to a two-layer perceptron model of a neural network, which functions as a neuron activation function. As a result, applying such a function aid in the modification of the prediction process in the same way as neural networks do [25]. The Naive Bayes (NB) method is a supervised classification algorithm that takes feature independence into account. It's very useful for datasets with a lot of features. The algorithm considers all features, even those with tiny effects on prediction. NB has received a lot of attention due to its simple classification model and excellent classification results [26].
Tables 4 and 5 show the parameter settings for the RF, SVM, GNB, and SHAP explainer configurations.
Table 4 Parameter setting for the algorithm.
Algorithm
|
Parameter setting
|
Definition
|
RF
|
n_estimators =100
|
The number of trees in the forest.
|
|
Criterion = entropy
|
The function to measure the quality of a split.
|
|
max_depth = none
|
The maximum depth of the tree
|
|
min_samples_split = 2
|
The minimum number of samples required to split an internal node
|
|
min_samples_leaf = 1
|
The minimum number of samples required to be at a leaf node.
|
|
min_weight_fraction_leaf = 0.0
|
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.
|
|
max_features = ”sqrt”
|
The number of features to consider when looking for the best split
|
SVM
|
Cfloat =1.0
|
The regularization parameter (lambda) determines the relevance of misclassifications.
|
|
Kernel: ‘linear’
|
The kernel method is a mathematical methodology used in machine learning to analyze data. Linear Kernel is used when the data is linearly separable.
|
|
degreeint =3
|
Polynomial kernel function degree ('poly'). It must not be negative.
|
|
random_state = none
|
The random state is a model hyperparameter that regulates the randomness in machine learning models. None: this enables the function to make use of the global random state instance.
|
GNB
|
priorsarray-like of shape (n_classes,) =None
|
The classes' prior probability. If this option is specified, the priors are not modified based on the data.
|
Table 5 SHAP explainer configurations.
Bar plot
|
Parameter setting
|
Definition
|
Global bar plot
|
shap.plots.bar(shap_values, max_display=28)
|
The global importance of each feature is defined as its mean absolute value across all samples. The features are sorted in this order based on their impact on the prediction. The bar plot displays a maximum of ten bars by default, however, this can be changed using the max_display parameter = 28 (number of feature predictions in this research).
|
Local bar plot
|
shap.plots.bar(shap_value[0])
|
This plot depicts the primary features influencing the prediction of a single observation, as well as the magnitude of the SHAP value for each feature.
|
e. Performance Measure
Afterwards, we evaluated our model using a variety of relative metrics. In our evaluation, we used a confusion matrix. These measures were calculated so far and were based on True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). If the dataset is not balanced, accuracy may not be a good measure [16]. We calculated the following measure scores: accuracy, precision, F1-score, and recall. We also computed the area under the receiver operating characteristic curve (AUC). The receiver operating characteristic (ROC) curve measures a classifier's prediction quality. The optimal position is thus in the upper left corner of the plot, where false positives equal 1 and genuine positives equal 0. The AUC is a measure or degree of separability. This demonstrates how well the model can distinguish between classes. A higher AUC indicates that the model more accurately predicts class 0 as 0 and class 1 as 1 [27].
f. Model-Agnostic Interpretation
ML models are typically viewed as black boxes that receive specific features and produce predictions. Approaches based on interpretability can help overcome the problems associated with black-box models. Although machine learning algorithms may learn complex associations and enhance forecast accuracy, their inner workings are complex. Interpretability approaches come into play here by providing a lens through which to view these complex models [28]. Fully complicated models can be interpreted either globally or locally using model-agnostic interpretability strategies. Global interpretability explains the model's overall behaviour throughout the entire population. Local interpretability, on the other hand, provides explanations for a given model prediction [29]. SHapley Additive exPlanations (SHAP) was employed for model interpretation. SHAP is widely used to interpret various classification and regression models. In this method, attributes are ranked according to their contribution to the model, and the relationship between attributes and results can be visualized. Its absolute value influences the effect of the attribute, and its positive or negative value reveals the attribute’s positive or negative impact on the prediction. A SHAP bar plot will use the mean absolute value of each attribute across all instances (rows) of the dataset by default [28]. SHAP presents a simple approach to f by explaining the contribution of each attribute value. Model g is as follows: (see Eq 1) [28]. In this case, p is the number of attributes, z = [z1,z2,…,zp] is a simplification in the input x, where z represents the data prediction attributes and is 1 and the unused attribute has a z value of 0. Furthermore, ∅i ∊ ℝ reflects each attribute's contribution to the model.
![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQYAAAA5CAYAAADUboXwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAABfaVRYdFNuaXBNZXRhZGF0YQAAAAAAeyJjbGlwUG9pbnRzIjpbeyJ4IjowLCJ5IjowfSx7IngiOjI2MiwieSI6MH0seyJ4IjoyNjIsInkiOjU3fSx7IngiOjAsInkiOjU3fV19zGIuvgAACGRJREFUeF7tncuPDF0Yh898e5LBSkgECzsLgwUWFi6xRUawsGNYSlwGKxuEjRhGYilBYiKREJeFBAv3EBErFmI5RoY/YL5+ztTL+U5XdVdVV/fUfPk9SaWrqk9Xncv7/s6lTp3um2rghBAi4J/kUwgh/iBhmAV8+PDBLV++3O3atcvdvHnTzZs3z61evTr5VojqkTDMAj5//uwuXrzoHj165CYnJ93ExIT7+fOne/78eRJCiGqRMMwCaCn8+vXLDQ0Nuf379/tziMPChQv9vhBVI2GYJTx9+tStW7fO/fjxwx08eNBt3rzZLV26NPlWiGqRMMwS6Ebs3bvXLViwwB+PjIz4TyG6gYRhFmCDj3QfeLp8+fJlN3/+/ORbIapHwjALePHihXv48KEGG0XP0AQnIUQTajEIIZqQMAghmpAwCCGakDDUhKtXr7q+vr5KtnPnziVXFaIcEoaasGPHDjcwMOD3+/v73fj4uH80mXcjvP1eiE6RMNQE5iVcu3bN7/MexJ49e/x+Xvj9rVu3kiMhOkPCUCNWrlzpRkdH/T7zFuheFIEp0suWLUuOhCiPhKFm8JLU4OCg3z9w4ICf9VgEZkgK0Sma4FRDeFFq7dq17suXL74F8PLlS02BFj1FLYYaggiMjY35fcTh1KlTfl+IXiFhqCmMN5w9e9bvX7lyxa/cJESv6Jow8MIPC4zk7SPz7J11Br5+/ZqcEUeOHHFbtmzx+8ob0VMYY6iaGzduTA0MDEyNj48nZ/Lx7NmzqUafeur9+/fJmd7TaLpPNZwxOZp5yMP+/n7GgXyedoMyZSXqBb4zODiYy3fMxvHTLCoXBiKIIceGRmQw7rRteHg4CfVXHHplqPfu3fOOYXHh3q0yrChcs1PIE4tfo3uRnC0HhoMBhekdGhpKvm0NYe134dYtwRL5aFURY99pFR1hKfessq9cGIjE6OhocvQXIohghIpGpDDSGBLZqQPkweJkQkDcOI7j2Qk4ThUgnuaICEUZEGfSRt6aEZmztxNDwsfh2O+liItmsipijq0CaNUCzvLXSoXBWgV8xmCMoVFxnKVyRBSD6zbcIxQgnM8UNk2wykB+VIW1bNIMIQ8IcWgkpJU0kwdcsxUYYFi7mEFWJaCiHFmObd0KyraVMFjlGNtTIavlRqZC5lThTXH8PI5gkUkTELCmcyvjtzDtNsKlQVr43uLAvaxJTPz4rgqqug5Yjc81ywgXvw3FmbLjmqS9VV7FWDzIJzFzUA6hDafRThggtgvI/VSCEfGNGzf6PztpGJKfdHP06FG3atWqJIRz3759+zOKngVPKVjU9O7du21XOeb/FLJYv349Htd2I1wav3//9p8Wh8ePH7vDhw/7/blz5/rPukFcr1+/7vfLvBfBOxiLFi3y+5Qn12MrMnmKyVcNUXLHjh1z27ZtS86KmeDVq1f+s50ftWPNmjXu48ePydE0uYWh0eT0F7CFSM2YWNI8LxjV9u3b/TWyHHamuHDhgtu0aVNyVA4e0cavQEN8rpO1G1esWOHfvmzUBMmZcpw/f97t3r07OcrPoUOH/L9g8ShVzCx5KuK8vH37NtmbJrcwUEPt27cvOZqu+YGpuyE4fxa8Mcj/ITC/oVPSnDBty3LCOXPm+E9qzvv377udO3f+ETv+3CXEVmnmeidOnEjONpPWioH4XCeiSG2NQJdxTATl+/fvvoxIt8UjrcxIbzwHhbkmjWarO336dHLmv3BNWpTif0DDUNti/fmwD0q/xfrkRqsxBgau4vBZA4x2v1ZjDFVgfSvSEvbT6L+HfXjiSZyID/tF+tY5szgXNkjYqk/ZCtJEOXCdsE/J4BXXbQXh43tzrbzjEqJ6KMfYp2LyjDHwfRwml9ViDBg4BoRzMJJtRhZi4WLDxahwqNDR04TCIDFZolEl5mgmAsTPztlouw1SGjaKn5eqhMHEMnTootg1wrw1hw+vS5hQGMmDME+A8ISzMrVrp42Qi+5gZdAK7LWdLyEKhAvJbbX8kEhwEQyEm6UZKd+HjmOOlbbFwmIgGEWcrxO4F2mxOOEQoQOYwRvEq9fCYC2V0FnLYoZi6SX9cQuI9JmD270tfLhxPsRsQ/SGrIoYKGfKNiyruFVg8H3sy22tFqOJ1SSuLUJwJGqXtO/ywP1IRNnfF4EMzcosw4TNIC96LQwIQlV5grG0u04sjnmJhUJ0n7giLgr+luavba2WG4c1lTl+LBYhCEceA4zBGDGuutU6pJcMJD3EjzzoFSbCvbwn6UU0i9yTMsNW+E1aDSa6g/lj2UoDP03r/rUVBgo8bpLkUSgiXKTmIXJ0LepoVDgnmU/6Wwli1ZAX3LeTGqEM3BMnL2JslHOv80dMU7Yixt+yuvNawanGMF8AXr9+7T+FyILH8pcuXXLHjx/3a3m0gsfTTB1g+kHW1AEJQ01hzsCZM2fcmzdvOp7ZJkRRJAw1BPXfsGGDXzGaxWGF6DUShppBM4/ZpI0+Y6nl3Fhy/s6dO+7BgwfJGSGKozUfawbvIkxMTLiRkZHkTDGePHmS7AlRHglDjaCFwDspvHla5I1Hgy4Iv9eYhOgUCUNN4AUkFnwFxhfil8HybPwOlixZ4j+FKIuEoSYMDw/79RKEqAMafBRCNKEWgxCiCQlDDWGsocg/Xdv4BJOihKgCCUMNYek7m9iEs6cNNJoIsMrS7du3/SNOIapCYww1Y+vWrX6Rz6KObkKhtRhFFajFUDNOnjzp18UUYiaRMNSMT58++WX6jXZdCSG6gYShZjClefHixX7laqBrQG8v3uIuw7t379zk5GRyJERnaIyhZvB+PK9aj42NtX2vHuxNzBAVqegUCYMQogl1JYQQTUgYhBBNSBiEEBHO/Qt7R+5Tm0PyUwAAAABJRU5ErkJggg==)
We used SHAP global and local interpretation to build a model-agnostic classification model for understanding predictive features. Global interpretation combines SHAP values across numerous instances to understand the general behaviour of a machine learning model and identify the most essential features influencing its predictions. Local interpretation focuses on understanding the causes influencing individual predictions using SHAP values, which provide instance-specific interpretations [28].