We can’t conclude our model with accuracy metrics alone. So, we evaluated the model with more metrics like precision recall, prediction error, learning curve, manifold learning, calibration curve, validation curve, lift chart, gain chart and ks plot.
5.1. PRECISION RECALL
The Precision Recall Curve illustrates the trade-off between recall, a measure of completeness, and precision, a measure of result relevance, in a classifier. Recall is defined as the ratio of true positives to the total of both true and false positives, whereas precision is the ratio of true positives to the total of both true and false positives for each class.
Precision: Precision is a metric for how accurate a classifier is. The ratio of true positives to the total of both true and false positives for each class is how it is defined.
Recall: A classifier's recall, or capacity to correctly identify every positive instance, is a gauge of how complete it is. It is described as the ratio for each class.
Both precision and recall have values between 0 and 1, and when choosing and fine-tuning machine learning models, we frequently aim to maximize both, producing a prototype that accurately identifies the majority of the classes it chooses. The result would be a Precision Recall Curve graphical analysis with a significant area under the curve.
The comparison of precision-recall for the machine learning algorithms that we compared in our research is shown in Fig. 14, where the average precision is 0.98 percent in the light gradient boosting machine compared to other algorithms.
5.2. PREDICTION ERROR
It assesses how well samples are classified into the appropriate category. The prediction problem is concerned with whether the samples are correctly categorised into their respective categories. The goal is to find a rule that predicts outcomes or categories well for new cases where the response or category is unknown.
The prediction error for the machine learning algorithms that we compared in our research is shown in Fig. 15, with the error being very low in the light gradient boosting machine when compared to other algorithms.
5.3. LEARNING CURVE
A learning curve is a graph that shows how the training and test accuracy scores change as the count of samples/rows in the data increases.
So you can see both the training and cross-validation scores on this curve. Increasing the number of examples has little effect on the training score. But the cross-validation score most emphatically does! You can see that once we get to about 1000–1200 examples, the performance changes very little. As a result, we can conclude that adding more examples to the ones we already have is unlikely to be necessary as shown in the Fig. 16.
5.4. MANIFOLD LEARNING
Manifold
A d-dimensional manifold is a region of an n-dimensional space (where d n) that resembles a d-dimensional hyperplane locally.
Creating a model of the manifold on which the training instances are located. It is predicated on numerous assumptions (manifold hypothesis). Most high-dimensional datasets in the real world are close to a much lower-dimensional manifold.
The learning curve for the machine learning algorithms compared in our research is shown in figure.17, with the classification being much more accurate to ransomware than normal permissions.
5.5. CALIBRATION CURVE
Calibration curves are used to assess how well a classifier is calibrated, i.e. how the probabilities of predicting for every class label vary. The estimated predicted probability in each bin is represented by the x-axis. The ratio of positives is represented by the y-axis (the proportion of positive predictions). The ideal calibrated model's curve is a linear direct line moving linearly from (0, 0).
The calibration curve for the machine learning algorithms compared in our research is shown in figure. 18, with the calibration in random forest tree classifiers for ransomware being much more accurate than normal permissions.
5.6. VALIDATION CURVE
The accuracy of a Machine Learning model's sensitivity to changes in some model parameters is demonstrated by a Validation Curve, a helpful diagnostic tool. The relationship between a model parameter and the model's score is typically represented by a validation curve. Two curves make up a validation curve: one for the cross-validation score and one for the training set score. The scikit-learn library's function for the validation curve must by default carry out 3-fold cross-validation.
The model's hyperparameters must be selected so that it can function in the designated feature space in order to maximize the score. A grid search is the most effective method for choosing a set of the multiple hyperparameters that are present in the majority of models. To determine whether the estimator is underfitting or overfitting for some hyper-parameter values, it can be helpful to plot the influence of a single hyperparameter on the training and testing data.
The validation curve for the machine learning algorithms compared in our study is shown in figure. 19, where random forest tree classifier, extra tree classifier, and light gradient boosting machine are much more accurate than normal permissions for ransomware.
5.7. LIFT CHAT
Lift is the proportion of positive observations using the model up to decile I to the positive observations predicted by the random model up to that decile I. An illustration of the relationship between the lift on the vertical axis and the corresponding decile on the horizontal axis is called a lift chart.
Figure 20, depicts the lift chart for the machine learning algorithms compared in our study, with decision tree classifiers showing more contrast to ransomware than normal permissions.
5.8. GAIN CHART
Gain is the proportion of all positive observational data to all cumulative positive analyses up to a decile. As seen in Fig. 21, the gain chart is a graph with the gain on the vertical axis and the decile on the horizontal axis.
5.9. KS STATISTIC PLOT
A variant of the ppcc plot is the Kolmogorov-Smirnov (or KS) plot. A ppcc plot is a graphical data processing technique used to determine which member of a given distributional family provides the "best" distributional fit of the model. The KS plot modifies the ppcc plot by replacing the correlation coefficient of the probability plot with the valuation of the Kolmogorov-Smirnov goodness of fit statistic as the way of measuring distributional fit. For the KS plot, we want to find the shape parameter value that minimises the Kolmogorov-Smirnov statistic. The KS plot is created by choosing a shape parameter and computing the Kolmogorov-Smirnov goodness of fit test value. The KS plot then includes the following:
The value of the distributional parameter (on the horizontal axis) that corresponds to the least of the KS plot curve (on the vertical axis) indicates the family member that fits the data the best.
The KS plot for the ml algorithms compared in our study is depicted in figure. 22, where all of the implemented machine learning algorithms show greater accuracy between 0.8 to 0.98 for normal and ransomware permissions.
5.10. COMPARISON WITH PRIOR ART
The performance of our suggested work is contrasted with a few other, comparable works in Table 1. This table demonstrates that, when compared to earlier available detection techniques, the identified structures of requested permissions and used characteristics offer an effective performance. The comparison table shows that the proposed method uses fewer features than other, comparable methods while still achieving a high detection accuracy.
Table 1
Performance comparison with similar work
Detection Methods | Type | Accuracy | Dataset | Balanced |
Alzahrani et al. [19] | Static/ Dynamic | 91% | 200 B 100 R | No |
Alsoghyer et al. [20] | Static | 97% | Not Specified | Yes |
Singh et al. [23] | Static | 93.92% | 1147 B 905 M | No |
Proposed Work | Static | 97.30% | 331 permissions dataset | Yes |