This section provides the predictive performance results in terms of accuracy, precision, recall, F1-score and AUC values obtained by modelling the proposed features using the basic supervised learning models and ensemble classifiers. Table 1 presents the classification accuracy, precision, and F1-score results achieved on different feature sets using typical supervised learning algorithms. Meanwhile, Table 2 displays the AUC and recall values obtained from conventional classifiers. Regarding the results of the proposed feature sets for classifying disaster response tweets using supervised learning algorithms, as presented in Table 1, it is evident that the logistic regression classifier typically outperforms the other classification algorithms in terms of classification accuracy and precision values. Among the five classifiers, the decision tree algorithm generally demonstrates the second-highest predictive performance, considering accuracy and precision values. The support vector machine algorithm follows in performance, and the random forest algorithm tends to achieve the lowest predictive accuracy. This experimental evaluation aims to assess the predictive capabilities of the proposed features, including TF-IDF, Hashtag, POS, and word embeddings such as GloVe, FastText, and Word2Vec, and their ensemble combinations when used as feature sets for classifying disaster response tweets. The empirical results presented in Table 1 and Table 2 reveal that, in the context of disaster response classification, the TF-IDF feature sets consistently outperform the POS features and other word embeddings such as Word2Vec (W2Vec), GloVe, FastText, and BERT. This superior performance is evident across various evaluation metrics, including accuracy, precision, recall, F1-score, and AUC, for all four classification algorithms employed in this study. In contrast, the hashtag features exhibit notably lower performance, recording the least accuracy, precision, recall, F1-score, and AUC across all the tested algorithms, as indicated in the aforementioned tables. Additionally, the analysis presented in Table 1 and Table 2 highlights that the three-word embeddings tested, namely GloVe, Word2Vec, and FastText, perform quite competitively across the four classification models investigated in this study. They demonstrate remarkably similar predictive performance concerning accuracy, precision, recall, F1-score, and AUC values. Hence, the experimental assessment of the suggested feature sets underscores the effectiveness of employing term frequency inverse document frequency (TF-IDF) features and word embeddings like GloVe, FastText, Word2Vec, and Bert, among others, for the classification of tweets into disaster and non-disaster categories. Notably, among the seven distinct feature sets, we attained the highest predictive accuracy, reaching 96.40 percent with the TF-IDF feature. As the second focus of our study, we delved into the combination of feature sets for disaster response classification. Tables 3–7 illustrate the application of ensemble feature sets, which involve the comnination of various proposed features with the aim of enhancing predictive performance in comparison to using individual feature sets. Notably, the ensemble feature set, which integrates all seven of the proposed feature sets, including TF-IDF, Hashtag, POS, FastText, Word2Vec, GloVe, and Bert, achieved the highest classification precision among the 31 feature sets that were taken into account. When this feature set was employed in conjunction with the random forest algorithm, it yielded an impressive classification accuracy of 98.92 percent and an F1-score of 96.50.
Table 1
Accuracy, Precision, and F1-Score results for the conventional classification model
| Accuracy values | Precision values | F1 SCORE values |
FEATURE SET | SVM | LR | RF | DT | SVM | LR | RF | DT | SVM | LR | RF | DT |
TF-IDF | 0.9535 | 0.9640 | 0.9590 | 0.9635 | 0.9714 | 0.9649 | 0.9860 | 0.9861 | 0.9562 | 0.9641 | 0.9611 | 0.9654 |
HASHTAG | 0.5620 | 0.5936 | 0.5637 | 0.5598 | 0.5524 | 0.7637 | 0.5533 | 0.9737 | 0.7104 | 0.5373 | 0.7111 | 0.3176 |
POS | 0.6539 | 0.6528 | 0.6678 | 0.6661 | 0.6686 | 0.6515 | 0.6767 | 0.6626 | 0.6895 | 0.6505 | 0.7053 | 0.7154 |
FAST TEXT | 0.8594 | 0.8594 | 0.8942 | 0.8056 | 0.8616 | 0.8594 | 0.9252 | 0.8047 | 0.8712 | 0.8591 | 0.8993 | 0.8244 |
GLOVE | 0.8588 | 0.8594 | 0.8893 | 0.8162 | 0.8600 | 0.8594 | 0.9235 | 0.8019 | 0.8709 | 0.8591 | 0.8942 | 0.8373 |
W2VEC | 0.8505 | 0.8522 | 0.8920 | 0.8311 | 0.8615 | 0.8521 | 0.9333 | 0.8215 | 0.8615 | 0.8521 | 0.8960 | 0.8488 |
BERT | 0.6168 | 0.6351 | 0.8245 | 0.7137 | 0.6329 | 0.6349 | 0.8500 | 0.7608 | 0.6608 | 0.6278 | 0.8345 | 0.7210 |
TF-IDF + HASH | 0.9546 | 0.9651 | 0.9635 | 0.9640 | 0.9735 | 0.9658 | 0.9861 | 0.9861 | 0.9572 | 0.9652 | 0.9654 | 0.9660 |
HASH + POS | 0.7016 | 0.7027 | 0.7060 | 0.6755 | 0.7202 | 0.7038 | 0.7114 | 0.7335 | 0.7257 | 0.7030 | 0.7378 | 0.6759 |
HAST + FST TEXT | 0.8671 | 0.8688 | 0.8876 | 0.8068 | 0.8754 | 0.8688 | 0.9223 | 0.8063 | 0.8772 | 0.8688 | 0.8925 | 0.8252 |
HASH + GLOVE | 0.8671 | 0.8660 | 0.8909 | 0.8217 | 0.8746 | 0.8660 | 0.9313 | 0.8192 | 0.8773 | 0.8660 | 0.8950 | 0.8388 |
HASH + W2VEC | 0.8555 | 0.8588 | 0.8887 | 0.8272 | 0.8688 | 0.8590 | 0.9281 | 0.8160 | 0.8657 | 0.8589 | 0.8930 | 0.8458 |
HASH + BERT | 0.6694 | 0.6772 | 0.8300 | 0.7159 | 0.6944 | 0.6789 | 0.8561 | 0.7880 | 0.6934 | 0.6776 | 0.8395 | 0.7113 |
POS + FST TEXT | 0.8837 | 0.8882 | 0.8859 | 0.8018 | 0.8791 | 0.8882 | 0.9193 | 0.8138 | 0.8942 | 0.8880 | 0.8911 | 0.8172 |
POS + GLOVE | 0.8837 | 0.8859 | 0.8909 | 0.8212 | 0.8814 | 0.8860 | 0.9275 | 0.8135 | 0.8938 | 0.8858 | 0.8955 | 0.8397 |
POS + W2VEC | 0.8649 | 0.8638 | 0.8893 | 0.8206 | 0.8726 | 0.8637 | 0.9291 | 0.8232 | 0.8753 | 0.8637 | 0.8935 | 0.8365 |
POS + BERT | 0.6573 | 0.6633 | 0.8322 | 0.7287 | 0.6773 | 0.6625 | 0.8613 | 0.7247 | 0.6872 | 0.6626 | 0.8409 | 0.7614 |
TF-IDF + POS | 0.9540 | 0.9668 | 0.9607 | 0.9635 | 0.9745 | 0.9675 | 0.9892 | 0.9861 | 0.9567 | 0.9668 | 0.9626 | 0.9654 |
TF-IDF + FST TEXT | 0.9507 | 0.9640 | 0.9258 | 0.9635 | 0.9703 | 0.9649 | 0.9586 | 0.9871 | 0.9536 | 0.9641 | 0.9292 | 0.9654 |
TF-IDF + GLOVE | 0.9529 | 0.9651 | 0.9252 | 0.9601 | 0.9744 | 0.9661 | 0.9605 | 0.9871 | 0.9556 | 0.9652 | 0.9285 | 0.9621 |
TF-IDF + W2VEC | 0.9524 | 0.9640 | 0.9247 | 0.9629 | 0.9724 | 0.9649 | 0.9646 | 0.9861 | 0.9551 | 0.9641 | 0.9276 | 0.9649 |
TF-IDF + BERT | 0.9535 | 0.9629 | 0.9540 | 0.9618 | 0.9744 | 0.9638 | 0.9880 | 0.9861 | 0.9561 | 0.9629 | 0.9561 | 0.9638 |
TF-IDF + HASH + POS | 0.9507 | 0.9646 | 0.9623 | 0.9635 | 0.9693 | 0.9652 | 0.9882 | 0.9861 | 0.9536 | 0.9646 | 0.9643 | 0.9654 |
TF-IDF + HASH + FATTEXT | 0.9529 | 0.9651 | 0.9247 | 0.9629 | 0.9714 | 0.9658 | 0.9595 | 0.9809 | 0.9557 | 0.9652 | 0.9280 | 0.9651 |
TF-IDF + HASH + GLOVE | 0.9518 | 0.9640 | 0.9208 | 0.9640 | 0.9713 | 0.9648 | 0.9581 | 0.9872 | 0.9546 | 0.9641 | 0.9241 | 0.9660 |
TF-IDF + HASH + W2VEC | 0.9546 | 0.9651 | 0.9203 | 0.9629 | 0.9755 | 0.9658 | 0.9581 | 0.9861 | 0.9572 | 0.9652 | 0.9235 | 0.9649 |
TF-IDF + HASH + BERT | 0.9529 | 0.9635 | 0.9574 | 0.9612 | 0.9734 | 0.9642 | 0.9891 | 0.9871 | 0.9556 | 0.9635 | 0.9593 | 0.9632 |
TF-IDF + HASH + POS + FAST TEXT | 0.9529 | 0.9646 | 0.9275 | 0.9635 | 0.9714 | 0.9652 | 0.9617 | 0.9871 | 0.9557 | 0.9646 | 0.9307 | 0.9654 |
TF-IDF + HASH + POS + GLOVE | 0.9518 | 0.9635 | 0.9280 | 0.9646 | 0.9713 | 0.9641 | 0.9638 | 0.9872 | 0.9546 | 0.9635 | 0.9311 | 0.9665 |
TF-IDF + HASH + POS + W2VEC | 0.9518 | 0.9646 | 0.9291 | 0.9629 | 0.9713 | 0.9652 | 0.9680 | 0.9861 | 0.9546 | 0.9646 | 0.9319 | 0.9649 |
TF-IDF + HASH + POS + BERT | 0.9535 | 0.9640 | 0.9546 | 0.9612 | 0.9734 | 0.9648 | 0.9890 | 0.9871 | 0.9562 | 0.9641 | 0.9566 | 0.9632 |
Table 2
AUC and RECALL performance results for the conventional classification model
| AUC values | RECALL values |
FEATURE SET | SVM | LR | RF | DT | SVM | LR | RF | DT |
TF-IDF | 0.9802 | 0.9654 | 0.9609 | 0.9650 | 0.9415 | 0.9640 | 0.9374 | 0.9456 |
HASHTAG | 0.6417 | 0.6227 | 0.5263 | 0.5919 | 0.9949 | 0.5936 | 0.9949 | 0.1897 |
POS | 0.6993 | 0.6466 | 0.6618 | 0.6565 | 0.7118 | 0.6528 | 0.7364 | 0.7774 |
FAST TEXT | 0.9267 | 0.8570 | 0.8959 | 0.8022 | 0.8810 | 0.8594 | 0.8749 | 0.8451 |
GLOVE | 0.9256 | 0.8571 | 0.8912 | 0.8110 | 0.8821 | 0.8594 | 0.8667 | 0.8759 |
W2VEC | 0.9224 | 0.8509 | 0.8947 | 0.8271 | 0.8615 | 0.8522 | 0.8615 | 0.8779 |
BERT | 0.6639 | 0.6244 | 0.8249 | 0.7162 | 0.6913 | 0.6351 | 0.8195 | 0.6851 |
TF-IDF + HASH | 0.9801 | 0.9664 | 0.9650 | 0.9655 | 0.9415 | 0.9651 | 0.9456 | 0.9467 |
HASH + POS | 0.7736 | 0.7022 | 0.7008 | 0.6798 | 0.7313 | 0.7027 | 0.7662 | 0.6267 |
HAST + FST TEXT | 0.9363 | 0.8679 | 0.8896 | 0.8034 | 0.8790 | 0.8688 | 0.8646 | 0.8451 |
HASH + GLOVE | 0.9353 | 0.8652 | 0.8935 | 0.8184 | 0.8800 | 0.8660 | 0.8615 | 0.8595 |
HASH + W2VEC | 0.9337 | 0.8583 | 0.8911 | 0.8228 | 0.8626 | 0.8588 | 0.8605 | 0.8779 |
HASH + BERT | 0.7572 | 0.6771 | 0.8306 | 0.7218 | 0.6923 | 0.6772 | 0.8236 | 0.6482 |
POS + FST TEXT | 0.9485 | 0.8862 | 0.8878 | 0.8001 | 0.9097 | 0.8882 | 0.8646 | 0.8205 |
POS + GLOVE | 0.9479 | 0.8840 | 0.8931 | 0.8171 | 0.9067 | 0.8859 | 0.8656 | 0.8677 |
POS + W2VEC | 0.9366 | 0.8626 | 0.8917 | 0.8180 | 0.8779 | 0.8638 | 0.8605 | 0.8503 |
POS + BERT | 0.7219 | 0.6594 | 0.8332 | 0.7223 | 0.6974 | 0.6633 | 0.8215 | 0.8021 |
TF-IDF + POS | 0.9796 | 0.9680 | 0.9627 | 0.9650 | 0.9395 | 0.9668 | 0.9374 | 0.9456 |
TF-IDF + FST TEXT | 0.9796 | 0.9654 | 0.9279 | 0.9651 | 0.9374 | 0.9640 | 0.9015 | 0.9446 |
TF-IDF + GLOVE | 0.9813 | 0.9666 | 0.9276 | 0.9620 | 0.9374 | 0.9651 | 0.8985 | 0.9385 |
TF-IDF + W2VEC | 0.9799 | 0.9654 | 0.9274 | 0.9645 | 0.9385 | 0.9640 | 0.8933 | 0.9446 |
TF-IDF + BERT | 0.9797 | 0.9643 | 0.9565 | 0.9635 | 0.9385 | 0.9629 | 0.9262 | 0.9426 |
TF-IDF + HASH + POS | 0.9781 | 0.9657 | 0.9642 | 0.9650 | 0.9385 | 0.9646 | 0.9415 | 0.9456 |
TF-IDF + HASH + FATTEXT | 0.9799 | 0.9664 | 0.9270 | 0.9640 | 0.9405 | 0.9651 | 0.8985 | 0.9497 |
TF-IDF + HASH + GLOVE | 0.9796 | 0.9653 | 0.9233 | 0.9656 | 0.9385 | 0.9640 | 0.8923 | 0.9456 |
TF-IDF + HASH + W2VEC | 0.9803 | 0.9664 | 0.9228 | 0.9645 | 0.9395 | 0.9651 | 0.8913 | 0.9446 |
TF-IDF + HASH + BERT | 0.9799 | 0.9647 | 0.9596 | 0.9630 | 0.9385 | 0.9635 | 0.9313 | 0.9405 |
TF-IDF + HASH + POS + FAST TEXT | 0.9797 | 0.9657 | 0.9297 | 0.9651 | 0.9405 | 0.9646 | 0.9015 | 0.9446 |
TF-IDF + HASH + POS + GLOVE | 0.9814 | 0.9646 | 0.9304 | 0.9661 | 0.9385 | 0.9635 | 0.9005 | 0.9467 |
TF-IDF + HASH + POS + W2VEC | 0.9803 | 0.9657 | 0.9318 | 0.9645 | 0.9385 | 0.9646 | 0.8985 | 0.9446 |
TF-IDF + HASH + POS + BERT | 0.9788 | 0.9653 | 0.9571 | 0.9630 | 0.9395 | 0.9640 | 0.9262 | 0.9405 |
To enhance the predictive capabilities of traditional supervised learning methods, we can employ the concept of ensemble learning. The third objective of our empirical analysis is to determine whether ensemble learners can achieve superior predictive results for disaster response classification. In this regard, we incorporated three well-established ensemble learners, namely bagging (B), AdaBoost (A), and random forest (RS). Tables 3–7 present the predictive performance metrics for ensemble learners when combined with standard learners and ensembles. As evident from the results presented in Tables 3–7, the employment of ensemble learning techniques generally led to improvements in the evaluation metrics compared to those achieved by standard classification algorithms. In terms of the outcomes obtained from different ensemble learning techniques, the random subspace algorithm consistently demonstrated the most remarkable predictive performance. Following this, the Bagging algorithm typically yielded the second-best predictive results, while the AdaBoost algorithm usually ranked third. The ensemble feature set that merged seven proposed feature sets, including TF-IDF, Hashtag, POS, and w2vec, in conjunction with the Bagging ensemble of decision trees, achieved the highest predictive accuracy among all comparative configurations. This setup reached a classification precision of 98.92 percent.
Table 3
Ensemble classification Accuracy performance values
FEATURE SET | A-SVM | A-LR | A-RF | A-DT | B-SVM | B-LR | B-RF | B-DT | RS-SVM | RS-LR | RS-RF | RS-DT |
TF-IDF | 0.9537 | 0.9635 | 0.9607 | 0.9618 | 0.9363 | 0.9568 | 0.9596 | 0.9618 | 0.5869 | 0.9081 | 0.9563 | 0.9579 |
HASHTAG | 0.5593 | 0.5681 | 0.5687 | 0.5399 | 0.5914 | 0.5925 | 0.5626 | 0.5399 | 0.5404 | 0.5548 | 0.5626 | 0.5421 |
POS | 0.6549 | 0.6689 | 0.6556 | 0.6423 | 0.6578 | 0.6495 | 0.6639 | 0.6539 | 0.6174 | 0.6312 | 0.6689 | 0.6501 |
FAST TEXT | 0.8364 | 0.8283 | 0.8527 | 0.8128 | 0.8533 | 0.8333 | 0.8865 | 0.8173 | 0.8245 | 0.7907 | 0.8854 | 0.8278 |
GLOVE | 0.8405 | 0.8355 | 0.8433 | 0.8267 | 0.8522 | 0.8372 | 0.8843 | 0.8084 | 0.8140 | 0.8389 | 0.8893 | 0.8173 |
W2VEC | 0.8376 | 0.8317 | 0.8488 | 0.8239 | 0.8450 | 0.8405 | 0.8865 | 0.8217 | 0.8350 | 0.8228 | 0.8848 | 0.8400 |
BERT | 0.7028 | 0.7724 | 0.7375 | 0.7281 | 0.6224 | 0.6373 | 0.8123 | 0.7132 | 0.5454 | 0.5853 | 0.8062 | 0.6960 |
TF-IDF + HASH | 0.9545 | 0.9640 | 0.9607 | 0.9618 | 0.9413 | 0.9601 | 0.9618 | 0.9618 | 0.5493 | 0.8843 | 0.9607 | 0.9646 |
HASH + POS | 0.6962 | 0.6932 | 0.6982 | 0.68 | 0.7043 | 0.6733 | 0.7049 | 0.66 | 0.5404 | 0.6772 | 0.7093 | 0.5393 |
HAST + FST TEXT | 0.8401 | 0.8339 | 0.8439 | 0.8206 | 0.8588 | 0.8295 | 0.8848 | 0.8228 | 0.7564 | 0.8212 | 0.8848 | 0.8173 |
HASH + GLOVE | 0.8430 | 0.835 | 0.8422 | 0.8311 | 0.8616 | 0.8499 | 0.8859 | 0.8173 | 0.7176 | 0.8505 | 0.8876 | 0.8007 |
HASH + W2VEC | 0.8433 | 0.8405 | 0.8522 | 0.8283 | 0.8527 | 0.8516 | 0.8865 | 0.8239 | 0.7586 | 0.8461 | 0.8843 | 0.8156 |
HASH + BERT | 0.6966 | 0.7674 | 0.7481 | 0.7237 | 0.6777 | 0.6373 | 0.8239 | 0.7132 | 0.5404 | 0.5759 | 0.8217 | 0.706 |
POS + FST TEXT | 0.8436 | 0.8311 | 0.8516 | 0.8278 | 0.8815 | 0.6633 | 0.8832 | 0.8223 | 0.8189 | 0.8128 | 0.8893 | 0.8256 |
POS + GLOVE | 0.8498 | 0.8555 | 0.8455 | 0.8256 | 0.8810 | 0.7769 | 0.8832 | 0.814 | 0.8106 | 0.8494 | 0.8848 | 0.8167 |
POS + W2VEC | 0.8499 | 0.8505 | 0.8538 | 0.8311 | 0.8621 | 0.6838 | 0.8843 | 0.8228 | 0.7580 | 0.8128 | 0.8843 | 0.8278 |
POS + BERT | 0.7084 | 0.7492 | 0.7453 | 0.7431 | 0.6650 | 0.6373 | 0.8245 | 0.7182 | 0.5382 | 0.6218 | 0.8178 | 0.6849 |
TF-IDF + POS | 0.9546 | 0.9646 | 0.9607 | 0.9618 | 0.9435 | 0.7946 | 0.9596 | 0.9618 | 0.5653 | 0.9064 | 0.9557 | 0.9601 |
TF-IDF + FST TEXT | 0.9561 | 0.9662 | 0.9629 | 0.9612 | 0.9385 | 0.9551 | 0.9197 | 0.9601 | 0.5991 | 0.9147 | 0.9158 | 0.9596 |
TF-IDF + GLOVE | 0.9577 | 0.9657 | 0.9635 | 0.9646 | 0.9419 | 0.9535 | 0.9186 | 0.964 | 0.6069 | 0.9452 | 0.9192 | 0.9585 |
TF-IDF + W2VEC | 0.9563 | 0.9623 | 0.9629 | 0.9618 | 0.9457 | 0.9557 | 0.9214 | 0.9579 | 0.6639 | 0.9186 | 0.9175 | 0.9646 |
TF-IDF + BERT | 0.8851 | 0.9607 | 0.9618 | 0.9618 | 0.9485 | 0.6373 | 0.9579 | 0.9607 | 0.5620 | 0.5670 | 0.9524 | 0.9612 |
TF-IDF + HASH + POS | 0.9546 | 0.9635 | 0.9607 | 0.9612 | 0.9468 | 0.8128 | 0.9623 | 0.9618 | 0.5853 | 0.9125 | 0.9601 | 0.9629 |
TF-IDF + HASH + FATTEXT | 0.9571 | 0.9618 | 0.9629 | 0.964 | 0.9369 | 0.9607 | 0.9208 | 0.9596 | 0.7004 | 0.9524 | 0.9147 | 0.9618 |
TF-IDF + HASH + GLOVE | 0.9579 | 0.9612 | 0.9635 | 0.9651 | 0.9424 | 0.9535 | 0.9219 | 0.9607 | 0.8106 | 0.9208 | 0.9158 | 0.9601 |
TF-IDF + HASH + W2VEC | 0.9570 | 0.9623 | 0.9629 | 0.9635 | 0.9413 | 0.9618 | 0.923 | 0.9557 | 0.6445 | 0.9546 | 0.9186 | 0.9563 |
TF-IDF + HASH + BERT | 0.8779 | 0.9574 | 0.9618 | 0.9635 | 0.9324 | 0.6373 | 0.9563 | 0.9601 | 0.5421 | 0.6207 | 0.9546 | 0.9551 |
TF-IDF + HASH + POS + FAST TEXT | 0.9550 | 0.9574 | 0.9629 | 0.9607 | 0.9480 | 0.8167 | 0.9258 | 0.9607 | 0.5460 | 0.9551 | 0.9142 | 0.9635 |
TF-IDF + HASH + POS + GLOVE | 0.9571 | 0.9618 | 0.9635 | 0.9646 | 0.9468 | 0.8560 | 0.9219 | 0.9612 | 0.6113 | 0.9496 | 0.9136 | 0.9574 |
TF-IDF + HASH + POS + W2VEC | 0.9577 | 0.9623 | 0.9629 | 0.9646 | 0.9441 | 0.8212 | 0.9269 | 0.9596 | 0.6921 | 0.9563 | 0.9169 | 0.959 |
TF-IDF + HASH + POS + BERT | 0.8732 | 0.9629 | 0.9618 | 0.9635 | 0.9391 | 0.6373 | 0.9557 | 0.9612 | 0.5471 | 0.6290 | 0.9557 | 0.9601 |
Table 4
Ensemble classification Precision performance values
FEATURE SET | A-SVM | A-LR | A-RF | A-DT | B-SVM | B-LR | B-DT | B-MLP | RS-SVM | RS-LR | RS-RF | RS-DT |
TF-IDF | 0.9688 | 0.9637 | 0.9809 | 0.9778 | 0.9715 | 0.9582 | 0.9892 | 0.9694 | 0.5696 | 0.9133 | 0.9848 | 0.9828 |
HASHTAG | 0.7061 | 0.7616 | 0.9712 | 0.5399 | 0.9611 | 0.7632 | 0.5399 | 0.9612 | 0.5402 | 0.7402 | 0.5527 | 0.5411 |
POS | 0.6598 | 0.6706 | 0.6712 | 0.629 | 0.6755 | 0.6485 | 0.6471 | 0.6728 | 0.6158 | 0.6296 | 0.6593 | 0.644 |
FAST TEXT | 0.8350 | 0.8282 | 0.8506 | 0.8095 | 0.8615 | 0.8332 | 0.8342 | 0.8515 | 0.8270 | 0.7935 | 0.9201 | 0.8395 |
GLOVE | 0.8396 | 0.8355 | 0.8446 | 0.8371 | 0.8590 | 0.8371 | 0.8117 | 0.8528 | 0.8111 | 0.8388 | 0.9282 | 0.8203 |
W2VEC | 0.8363 | 0.8316 | 0.8531 | 0.8211 | 0.8571 | 0.8407 | 0.8275 | 0.857 | 0.8423 | 0.8232 | 0.9247 | 0.8389 |
BERT | 0.7401 | 0.7742 | 0.758 | 0.7474 | 0.6441 | 0.6380 | 0.7853 | 0.6698 | 0.5522 | 0.5818 | 0.8307 | 0.7399 |
TF-IDF + HASH | 0.9702 | 0.9649 | 0.9809 | 0.9778 | 0.9844 | 0.9616 | 0.9892 | 0.9653 | 0.5456 | 0.8901 | 0.9871 | 0.975 |
HASH + POS | 0.7337 | 0.7046 | 0.7535 | 0.7338 | 0.7294 | 0.6723 | 0.6576 | 0.75 | 0.5402 | 0.6765 | 0.7023 | 0.5396 |
HAST + FST TEXT | 0.8455 | 0.8338 | 0.8434 | 0.8352 | 0.8704 | 0.8293 | 0.8408 | 0.8603 | 0.7570 | 0.8213 | 0.9155 | 0.8479 |
HASH + GLOVE | 0.8433 | 0.8349 | 0.8457 | 0.8357 | 0.8718 | 0.8502 | 0.8215 | 0.8652 | 0.7127 | 0.8504 | 0.9232 | 0.8135 |
HASH + W2VEC | 0.8545 | 0.8406 | 0.8598 | 0.8556 | 0.8681 | 0.8515 | 0.8308 | 0.8512 | 0.7604 | 0.8461 | 0.9246 | 0.8296 |
HASH + BERT | 0.7469 | 0.7699 | 0.7692 | 0.7742 | 0.7106 | 0.6380 | 0.7853 | 0.716 | 0.5403 | 0.5776 | 0.8492 | 0.7775 |
POS + FST TEXT | 0.8429 | 0.8312 | 0.8476 | 0.8211 | 0.8801 | 0.6624 | 0.8337 | 0.8533 | 0.8152 | 0.8162 | 0.9207 | 0.8395 |
POS + GLOVE | 0.8478 | 0.8555 | 0.8446 | 0.8198 | 0.8785 | 0.7793 | 0.8154 | 0.8666 | 0.8240 | 0.8493 | 0.9238 | 0.8411 |
POS + W2VEC | 0.8586 | 0.8508 | 0.8631 | 0.8425 | 0.8712 | 0.6833 | 0.8359 | 0.8548 | 0.7690 | 0.8133 | 0.9265 | 0.8409 |
POS + BERT | 0.7475 | 0.7495 | 0.7593 | 0.7687 | 0.6850 | 0.6380 | 0.719 | 0.6736 | 0.5402 | 0.6201 | 0.8443 | 0.7197 |
TF-IDF + POS | 0.9706 | 0.9655 | 0.9809 | 0.9809 | 0.9750 | 0.7956 | 0.9892 | 0.9662 | 0.5552 | 0.9128 | 0.9848 | 0.9758 |
TF-IDF + FST TEXT | 0.9735 | 0.9667 | 0.984 | 0.9881 | 0.9665 | 0.9567 | 0.9892 | 0.9676 | 0.5770 | 0.9190 | 0.9478 | 0.9892 |
TF-IDF + GLOVE | 0.9726 | 0.966 | 0.9851 | 0.98 | 0.9688 | 0.9550 | 0.9882 | 0.9553 | 0.6003 | 0.9469 | 0.951 | 0.9828 |
TF-IDF + W2VEC | 0.9734 | 0.9631 | 0.984 | 0.985 | 0.9710 | 0.9572 | 0.9891 | 0.9646 | 0.6617 | 0.9231 | 0.9538 | 0.9872 |
TF-IDF + BERT | 0.9013 | 0.962 | 0.985 | 0.9799 | 0.9804 | 0.6380 | 0.9839 | 0.6856 | 0.5558 | 0.6034 | 0.9879 | 0.9829 |
TF-IDF + HASH + POS | 0.9675 | 0.9638 | 0.9809 | 0.9778 | 0.9793 | 0.8128 | 0.9892 | 0.9656 | 0.5678 | 0.9162 | 0.9871 | 0.985 |
TF-IDF + HASH + FATTEXT | 0.9716 | 0.9628 | 0.984 | 0.9779 | 0.9674 | 0.9621 | 0.9892 | 0.9689 | 0.7039 | 0.9546 | 0.9467 | 0.9829 |
TF-IDF + HASH + GLOVE | 0.9712 | 0.9616 | 0.9851 | 0.982 | 0.9749 | 0.9547 | 0.9881 | 0.9658 | 0.9084 | 0.9219 | 0.9537 | 0.9819 |
TF-IDF + HASH + W2VEC | 0.9720 | 0.9631 | 0.984 | 0.984 | 0.9697 | 0.9631 | 0.9891 | 0.9647 | 0.6439 | 0.9563 | 0.9559 | 0.9786 |
TF-IDF + HASH + BERT | 0.9156 | 0.9579 | 0.985 | 0.9769 | 0.9765 | 0.6380 | 0.9839 | 0.6508 | 0.6968 | 0.6233 | 0.989 | 0.9806 |
TF-IDF + HASH + POS + FAST TEXT | 0.9688 | 0.9579 | 0.984 | 0.9738 | 0.9762 | 0.8166 | 0.9892 | 0.967 | 0.5434 | 0.9572 | 0.9515 | 0.9851 |
TF-IDF + HASH + POS + GLOVE | 0.9726 | 0.9625 | 0.9851 | 0.982 | 0.9741 | 0.8560 | 0.9881 | 0.967 | 0.6174 | 0.9513 | 0.9495 | 0.9828 |
TF-IDF + HASH + POS + W2VEC | 0.9725 | 0.9631 | 0.984 | 0.9841 | 0.9740 | 0.8210 | 0.9892 | 0.9672 | 0.6801 | 0.9582 | 0.9538 | 0.9798 |
TF-IDF + HASH + POS + BERT | 0.8766 | 0.9637 | 0.985 | 0.9769 | 0.9716 | 0.6380 | 0.984 | 0.7307 | 0.5445 | 0.6816 | 0.9891 | 0.9768 |
Table 5
Ensemble classification Recall performance values
FEATURE SET | A-SVM | A-LR | A-RF | A-DT | B-SVM | B-LR | B-RF | B-DT | RS-SVM | RS-LR | RS-RF | RS-DT |
TF-IDF | 0.9432 | 0.9635 | 0.9456 | 0.9508 | 0.9087 | 0.9568 | 0.9364 | 0.9395 | 0.9610 | 0.9081 | 0.9333 | 0.9385 |
HASHTAG | 0.6910 | 0.5681 | 0.2072 | 1.0000 | 0.2533 | 0.5925 | 0.9928 | 1.0000 | 1.0000 | 0.5548 | 0.9949 | 1.0000 |
POS | 0.7275 | 0.6689 | 0.7097 | 0.8226 | 0.7046 | 0.6495 | 0.7333 | 0.7897 | 0.7744 | 0.6312 | 0.8000 | 0.7867 |
FAST TEXT | 0.8607 | 0.8283 | 0.8821 | 0.8544 | 0.8677 | 0.8333 | 0.8656 | 0.8256 | 0.8533 | 0.7907 | 0.8626 | 0.8421 |
GLOVE | 0.8635 | 0.8355 | 0.8697 | 0.8431 | 0.8687 | 0.8372 | 0.8564 | 0.84 | 0.8544 | 0.8389 | 0.8615 | 0.8472 |
W2VEC | 0.8618 | 0.8317 | 0.8697 | 0.8615 | 0.8554 | 0.8405 | 0.8585 | 0.8462 | 0.8544 | 0.8228 | 0.8564 | 0.8708 |
BERT | 0.6677 | 0.7724 | 0.7549 | 0.7497 | 0.6718 | 0.6373 | 0.8031 | 0.6451 | 0.8359 | 0.5853 | 0.8051 | 0.6738 |
TF-IDF + HASH | 0.9436 | 0.964 | 0.9456 | 0.9508 | 0.9056 | 0.9601 | 0.9426 | 0.9395 | 0.9887 | 0.8843 | 0.9395 | 0.959 |
HASH + POS | 0.6761 | 0.6932 | 0.6554 | 0.639 | 0.7190 | 0.6733 | 0.7641 | 0.7723 | 0.9990 | 0.6772 | 0.801 | 0.999 |
HAST + FST TEXT | 0.8536 | 0.8339 | 0.8728 | 0.8318 | 0.8677 | 0.8295 | 0.8626 | 0.8287 | 0.8082 | 0.8212 | 0.8667 | 0.8062 |
HASH + GLOVE | 0.8634 | 0.835 | 0.8656 | 0.8554 | 0.8718 | 0.8499 | 0.8626 | 0.8451 | 0.7990 | 0.8505 | 0.8636 | 0.8185 |
HASH + W2VEC | 0.8483 | 0.8405 | 0.8677 | 0.8205 | 0.8574 | 0.8516 | 0.8574 | 0.8462 | 0.8072 | 0.8461 | 0.8554 | 0.8287 |
HASH + BERT | 0.6326 | 0.7674 | 0.7621 | 0.6892 | 0.6800 | 0.6373 | 0.8236 | 0.6451 | 0.9959 | 0.5759 | 0.8144 | 0.6379 |
POS + FST TEXT | 0.8657 | 0.8311 | 0.8841 | 0.8708 | 0.9036 | 0.6633 | 0.8615 | 0.8379 | 0.8595 | 0.8128 | 0.8697 | 0.8369 |
POS + GLOVE | 0.8736 | 0.8555 | 0.8749 | 0.8677 | 0.9046 | 0.7769 | 0.8533 | 0.8472 | 0.8256 | 0.8494 | 0.8574 | 0.8144 |
POS + W2VEC | 0.8580 | 0.8505 | 0.8667 | 0.8451 | 0.8738 | 0.6838 | 0.8513 | 0.8359 | 0.7887 | 0.8128 | 0.8533 | 0.84 |
POS + BERT | 0.6734 | 0.7492 | 0.7733 | 0.7497 | 0.7026 | 0.6373 | 0.8195 | 0.7846 | 0.9723 | 0.6218 | 0.8123 | 0.6821 |
TF-IDF + POS | 0.9435 | 0.9646 | 0.9456 | 0.9477 | 0.9190 | 0.7946 | 0.9374 | 0.9395 | 0.9795 | 0.9064 | 0.9323 | 0.9497 |
TF-IDF + FST TEXT | 0.9434 | 0.9662 | 0.9467 | 0.9395 | 0.9179 | 0.9551 | 0.8933 | 0.9364 | 0.9641 | 0.9147 | 0.8933 | 0.9354 |
TF-IDF + GLOVE | 0.9471 | 0.9657 | 0.9467 | 0.9538 | 0.9221 | 0.9535 | 0.8913 | 0.9446 | 0.8133 | 0.9452 | 0.8964 | 0.9395 |
TF-IDF + W2VEC | 0.9437 | 0.9623 | 0.9467 | 0.9436 | 0.9272 | 0.9557 | 0.8903 | 0.9323 | 0.7723 | 0.9186 | 0.8903 | 0.9467 |
TF-IDF + BERT | 0.8861 | 0.9607 | 0.9436 | 0.9487 | 0.9231 | 0.6373 | 0.9313 | 0.9426 | 0.9405 | 0.5670 | 0.9231 | 0.9446 |
TF-IDF + HASH + POS | 0.9465 | 0.9635 | 0.9456 | 0.9497 | 0.9210 | 0.8128 | 0.9415 | 0.9395 | 0.9703 | 0.9125 | 0.9385 | 0.9456 |
TF-IDF + HASH + FATTEXT | 0.9471 | 0.9618 | 0.9467 | 0.9549 | 0.9138 | 0.9607 | 0.8964 | 0.9354 | 0.7682 | 0.9524 | 0.8923 | 0.9456 |
TF-IDF + HASH + GLOVE | 0.9490 | 0.9612 | 0.9467 | 0.9528 | 0.9169 | 0.9535 | 0.8944 | 0.9385 | 0.7221 | 0.9208 | 0.8872 | 0.9436 |
TF-IDF + HASH + W2VEC | 0.9465 | 0.9623 | 0.9467 | 0.9477 | 0.9200 | 0.9618 | 0.8903 | 0.9282 | 0.7641 | 0.9546 | 0.8903 | 0.9395 |
TF-IDF + HASH + BERT | 0.8337 | 0.9574 | 0.9436 | 0.9549 | 0.8964 | 0.6373 | 0.9282 | 0.9415 | 0.2687 | 0.6207 | 0.9262 | 0.9354 |
TF-IDF + HASH + POS + FAST TEXT | 0.9458 | 0.9574 | 0.9467 | 0.9528 | 0.9262 | 0.8167 | 0.9005 | 0.9374 | 0.9959 | 0.9551 | 0.8862 | 0.9467 |
TF-IDF + HASH + POS + GLOVE | 0.9461 | 0.9618 | 0.9467 | 0.9518 | 0.9262 | 0.8560 | 0.8964 | 0.9395 | 0.7364 | 0.9496 | 0.8872 | 0.9374 |
TF-IDF + HASH + POS + W2VEC | 0.9473 | 0.9623 | 0.9467 | 0.9497 | 0.9210 | 0.8212 | 0.8954 | 0.9354 | 0.8113 | 0.9563 | 0.8892 | 0.9436 |
TF-IDF + HASH + POS + BERT | 0.9564 | 0.9629 | 0.9436 | 0.9549 | 0.9138 | 0.6373 | 0.9282 | 0.9436 | 0.9846 | 0.6290 | 0.9282 | 0.9487 |
Table 6
Ensemble classification F1-score performance values
FEATURE SET | A-SVM | A-LR | A-RF | A-DT | B-SVM | B-LR | B-RF | B-DT | RS-SVM | RS-LR | RS-RF | RS-DT |
TF-IDF | 0.9557 | 0.9635 | 0.9629 | 0.9641 | 0.9391 | 0.9569 | 0.9616 | 0.9637 | 0.7153 | 0.9082 | 0.9584 | 0.9601 |
HASHTAG | 0.5618 | 0.496 | 0.3415 | 0.7012 | 0.4010 | 0.5356 | 0.7102 | 0.7012 | 0.7014 | 0.4121 | 0.7106 | 0.7022 |
POS | 0.6884 | 0.6627 | 0.6899 | 0.7129 | 0.6898 | 0.6455 | 0.702 | 0.7113 | 0.6861 | 0.6289 | 0.7229 | 0.7082 |
FAST TEXT | 0.8476 | 0.8282 | 0.8661 | 0.8313 | 0.8646 | 0.8332 | 0.8917 | 0.8299 | 0.8400 | 0.7887 | 0.8904 | 0.8408 |
GLOVE | 0.8511 | 0.8353 | 0.857 | 0.8401 | 0.8638 | 0.8371 | 0.8888 | 0.8256 | 0.8322 | 0.8386 | 0.8936 | 0.8335 |
W2VEC | 0.8488 | 0.8317 | 0.8614 | 0.8408 | 0.8563 | 0.8401 | 0.8909 | 0.8367 | 0.8483 | 0.8222 | 0.8892 | 0.8546 |
BERT | 0.6942 | 0.7728 | 0.7564 | 0.7486 | 0.6576 | 0.6289 | 0.822 | 0.7083 | 0.6650 | 0.5774 | 0.8177 | 0.7053 |
TF-IDF + HASH | 0.9565 | 0.9641 | 0.9629 | 0.9641 | 0.9434 | 0.9602 | 0.9638 | 0.9637 | 0.7031 | 0.8844 | 0.9627 | 0.9669 |
HASH + POS | 0.7017 | 0.6929 | 0.701 | 0.6831 | 0.7242 | 0.6714 | 0.7365 | 0.7104 | 0.7012 | 0.6766 | 0.7484 | 0.7007 |
HAST + FST TEXT | 0.8495 | 0.8338 | 0.8579 | 0.8335 | 0.8690 | 0.8293 | 0.8899 | 0.8347 | 0.7817 | 0.8207 | 0.8904 | 0.8265 |
HASH + GLOVE | 0.8532 | 0.835 | 0.8555 | 0.8454 | 0.8718 | 0.8500 | 0.8909 | 0.8332 | 0.7534 | 0.8504 | 0.8924 | 0.816 |
HASH + W2VEC | 0.8513 | 0.8406 | 0.8637 | 0.8377 | 0.8627 | 0.8514 | 0.8908 | 0.8384 | 0.7831 | 0.8458 | 0.8887 | 0.8291 |
HASH + BERT | 0.6723 | 0.7678 | 0.7656 | 0.7292 | 0.6950 | 0.6289 | 0.8347 | 0.7083 | 0.7006 | 0.5398 | 0.8314 | 0.7008 |
POS + FST TEXT | 0.8540 | 0.8312 | 0.8655 | 0.8452 | 0.8917 | 0.6606 | 0.8884 | 0.8358 | 0.8367 | 0.8111 | 0.8945 | 0.8382 |
POS + GLOVE | 0.8604 | 0.8553 | 0.8594 | 0.843 | 0.8914 | 0.7747 | 0.8875 | 0.831 | 0.8248 | 0.8492 | 0.8894 | 0.8275 |
POS + W2VEC | 0.8583 | 0.8506 | 0.8649 | 0.8438 | 0.8725 | 0.6814 | 0.8882 | 0.8359 | 0.7787 | 0.8121 | 0.8884 | 0.8404 |
POS + BERT | 0.7011 | 0.7493 | 0.7663 | 0.7591 | 0.6937 | 0.6289 | 0.8345 | 0.7504 | 0.6945 | 0.6162 | 0.828 | 0.7004 |
TF-IDF + POS | 0.9567 | 0.9646 | 0.9629 | 0.964 | 0.9461 | 0.7934 | 0.9616 | 0.9637 | 0.7087 | 0.9065 | 0.9579 | 0.9626 |
TF-IDF + FST TEXT | 0.9581 | 0.9663 | 0.965 | 0.9632 | 0.9416 | 0.9552 | 0.9232 | 0.9621 | 0.7220 | 0.9149 | 0.9197 | 0.9615 |
TF-IDF + GLOVE | 0.9596 | 0.9657 | 0.9655 | 0.9667 | 0.9448 | 0.9536 | 0.922 | 0.9659 | 0.6908 | 0.9453 | 0.9229 | 0.9607 |
TF-IDF + W2VEC | 0.9582 | 0.9624 | 0.965 | 0.9639 | 0.9486 | 0.9558 | 0.9244 | 0.9599 | 0.7127 | 0.9187 | 0.921 | 0.9665 |
TF-IDF + BERT | 0.8933 | 0.9607 | 0.9639 | 0.964 | 0.9509 | 0.6289 | 0.9598 | 0.9628 | 0.6987 | 0.4749 | 0.9544 | 0.9634 |
TF-IDF + HASH + POS | 0.9568 | 0.9635 | 0.9629 | 0.9636 | 0.9493 | 0.8124 | 0.9643 | 0.9637 | 0.7164 | 0.9127 | 0.9621 | 0.9649 |
TF-IDF + HASH + FATTEXT | 0.9590 | 0.9618 | 0.965 | 0.9663 | 0.9399 | 0.9607 | 0.9244 | 0.9615 | 0.7347 | 0.9525 | 0.9187 | 0.9639 |
TF-IDF + HASH + GLOVE | 0.9599 | 0.9613 | 0.9655 | 0.9672 | 0.9450 | 0.9536 | 0.9252 | 0.9627 | 0.8046 | 0.9209 | 0.9192 | 0.9623 |
TF-IDF + HASH + W2VEC | 0.9589 | 0.9624 | 0.965 | 0.9655 | 0.9442 | 0.9618 | 0.9259 | 0.9577 | 0.6989 | 0.9547 | 0.9219 | 0.9587 |
TF-IDF + HASH + BERT | 0.8674 | 0.9574 | 0.9639 | 0.9658 | 0.9348 | 0.6289 | 0.9582 | 0.9623 | 0.3879 | 0.6062 | 0.9566 | 0.9575 |
TF-IDF + HASH + POS + FAST TEXT | 0.9571 | 0.9574 | 0.965 | 0.9632 | 0.9505 | 0.8165 | 0.9291 | 0.9626 | 0.7031 | 0.9552 | 0.9177 | 0.9655 |
TF-IDF + HASH + POS + GLOVE | 0.9590 | 0.9618 | 0.9655 | 0.9667 | 0.9495 | 0.8560 | 0.9254 | 0.9632 | 0.6717 | 0.9497 | 0.9173 | 0.9596 |
TF-IDF + HASH + POS + W2VEC | 0.9596 | 0.9624 | 0.965 | 0.9666 | 0.9468 | 0.8210 | 0.9297 | 0.9615 | 0.7399 | 0.9563 | 0.9204 | 0.9613 |
TF-IDF + HASH + POS + BERT | 0.9043 | 0.9629 | 0.9639 | 0.9658 | 0.9419 | 0.6289 | 0.9577 | 0.9634 | 0.7012 | 0.5800 | 0.9577 | 0.9625 |
Table 7
Ensemble classification AUC performance values
FEATURE SET | A-SVM | A-LR | A-RF | A-DT | B-SVM | B-LR | B-RF | B-DT | RS-SVM | RS-LR | RS-RF | RS-DT |
TF-IDF | 0.9547 | 0.9641 | 0.962 | 0.9627 | 0.9807 | 0.9586 | 0.9616 | 0.9637 | 0.5781 | 0.9117 | 0.9582 | 0.9596 |
HASHTAG | 0.5557 | 0.5995 | 0.6 | 0.5 | 0.6364 | 0.6217 | 0.5253 | 0.5 | 0.5026 | 0.5163 | 0.5251 | 0.5024 |
POS | 0.6461 | 0.6587 | 0.6509 | 0.6267 | 0.7033 | 0.6414 | 0.6579 | 0.6422 | 0.6035 | 0.6250 | 0.6575 | 0.6382 |
FAST TEXT | 0.8338 | 0.8265 | 0.8502 | 0.8092 | 0.9252 | 0.8318 | 0.8883 | 0.8166 | 0.8970 | 0.7842 | 0.8874 | 0.8266 |
GLOVE | 0.8379 | 0.8331 | 0.841 | 0.8253 | 0.9246 | 0.8354 | 0.8867 | 0.8057 | 0.8850 | 0.8366 | 0.8917 | 0.8147 |
W2VEC | 0.8353 | 0.8305 | 0.847 | 0.8207 | 0.9208 | 0.8376 | 0.8889 | 0.8196 | 0.9077 | 0.8192 | 0.8873 | 0.8373 |
BERT | 0.7060 | 0.7731 | 0.736 | 0.7263 | 0.6649 | 0.6258 | 0.8131 | 0.7191 | 0.5296 | 0.5745 | 0.8063 | 0.6979 |
TF-IDF + HASH | 0.9558 | 0.9654 | 0.962 | 0.9627 | 0.9840 | 0.9620 | 0.9635 | 0.9637 | 0.5458 | 0.8880 | 0.9625 | 0.965 |
HASH + POS | 0.6993 | 0.6987 | 0.7019 | 0.6835 | 0.7743 | 0.6676 | 0.6997 | 0.6503 | 0.5017 | 0.6737 | 0.7014 | 0.4995 |
HAST + FST TEXT | 0.8386 | 0.8326 | 0.8413 | 0.8196 | 0.9338 | 0.8274 | 0.8868 | 0.8223 | 0.8087 | 0.8179 | 0.8864 | 0.8182 |
HASH + GLOVE | 0.8410 | 0.8338 | 0.8402 | 0.829 | 0.9338 | 0.8496 | 0.888 | 0.8149 | 0.7612 | 0.8492 | 0.8897 | 0.7991 |
HASH + W2VEC | 0.8427 | 0.8398 | 0.8508 | 0.829 | 0.9299 | 0.8497 | 0.889 | 0.822 | 0.8297 | 0.8437 | 0.8868 | 0.8145 |
HASH + BERT | 0.7024 | 0.7686 | 0.7468 | 0.7267 | 0.7564 | 0.6258 | 0.8239 | 0.7191 | 0.5928 | 0.5539 | 0.8223 | 0.7119 |
POS + FST TEXT | 0.8415 | 0.8303 | 0.8488 | 0.8241 | 0.9468 | 0.6565 | 0.885 | 0.8209 | 0.8885 | 0.8065 | 0.8909 | 0.8246 |
POS + GLOVE | 0.8472 | 0.8533 | 0.843 | 0.8219 | 0.9472 | 0.7702 | 0.8858 | 0.8111 | 0.8891 | 0.8474 | 0.8872 | 0.8169 |
POS + W2VEC | 0.8492 | 0.8503 | 0.8527 | 0.8299 | 0.9355 | 0.6773 | 0.8871 | 0.8217 | 0.8253 | 0.8089 | 0.887 | 0.8267 |
POS + BERT | 0.7111 | 0.7481 | 0.7429 | 0.7425 | 0.7253 | 0.6258 | 0.8249 | 0.7124 | 0.5707 | 0.6124 | 0.8183 | 0.6852 |
TF-IDF + POS | 0.9559 | 0.966 | 0.962 | 0.963 | 0.9812 | 0.7896 | 0.9615 | 0.9637 | 0.5556 | 0.9105 | 0.9577 | 0.961 |
TF-IDF + FST TEXT | 0.9574 | 0.9671 | 0.9643 | 0.9631 | 0.9832 | 0.9570 | 0.922 | 0.9622 | 0.5654 | 0.9179 | 0.9178 | 0.9617 |
TF-IDF + GLOVE | 0.9588 | 0.9664 | 0.9649 | 0.9655 | 0.9831 | 0.9553 | 0.921 | 0.9657 | 0.5957 | 0.9471 | 0.9211 | 0.9601 |
TF-IDF + W2VEC | 0.9577 | 0.9636 | 0.9643 | 0.9634 | 0.9817 | 0.9576 | 0.9241 | 0.9601 | 0.6934 | 0.9219 | 0.9199 | 0.9661 |
TF-IDF + BERT | 0.8855 | 0.9624 | 0.9634 | 0.9629 | 0.9828 | 0.6258 | 0.9602 | 0.9623 | 0.5903 | 0.5343 | 0.9549 | 0.9627 |
TF-IDF + HASH + POS | 0.9555 | 0.9642 | 0.962 | 0.9622 | 0.9835 | 0.8100 | 0.9642 | 0.9637 | 0.5990 | 0.9154 | 0.962 | 0.9644 |
TF-IDF + HASH + FATTEXT | 0.9583 | 0.9633 | 0.9643 | 0.9648 | 0.9824 | 0.9625 | 0.9229 | 0.9617 | 0.7458 | 0.9547 | 0.9167 | 0.9632 |
TF-IDF + HASH + GLOVE | 0.9589 | 0.9621 | 0.9649 | 0.9662 | 0.9831 | 0.9551 | 0.9243 | 0.9626 | 0.9046 | 0.9220 | 0.9183 | 0.9616 |
TF-IDF + HASH + W2VEC | 0.9582 | 0.9636 | 0.9643 | 0.9648 | 0.9820 | 0.9635 | 0.9259 | 0.9581 | 0.6815 | 0.9566 | 0.9211 | 0.9577 |
TF-IDF + HASH + BERT | 0.8820 | 0.9583 | 0.9634 | 0.9642 | 0.9809 | 0.6258 | 0.9587 | 0.9617 | 0.6240 | 0.6059 | 0.9571 | 0.9569 |
TF-IDF + HASH + POS + FAST TEXT | 0.9561 | 0.9584 | 0.9643 | 0.9614 | 0.9830 | 0.8145 | 0.928 | 0.9627 | 0.5457 | 0.9574 | 0.9166 | 0.9649 |
TF-IDF + HASH + POS + GLOVE | 0.9584 | 0.963 | 0.9649 | 0.9657 | 0.9822 | 0.8547 | 0.9241 | 0.9631 | 0.6535 | 0.9516 | 0.9159 | 0.9591 |
TF-IDF + HASH + POS + W2VEC | 0.9589 | 0.9636 | 0.9643 | 0.9658 | 0.9825 | 0.8192 | 0.9296 | 0.9617 | 0.7030 | 0.9584 | 0.9193 | 0.9604 |
TF-IDF + HASH + POS + BERT | 0.8663 | 0.9642 | 0.9634 | 0.9642 | 0.9812 | 0.6258 | 0.9581 | 0.9628 | 0.5332 | 0.6026 | 0.9581 | 0.9611 |