Demographic and clinicopathological features
In this study, 256 patients were enrolled, among which 81 were pathologically confirmed to have lymph node metastasis post-surgery, accounting for 31.6% of all patients. Significant differences were observed between the group with cervical lymph node metastasis (N1 group) and the group without cervical lymph node metastasis (N0 group) regarding maximum tumor diameter, number of lesions, and ultrasound imaging ACR score. However, age and gender did not show a statistical difference between the two groups according to the Chi-squared test analysis. Detailed information is described in Table 1.
Table 1
The demographic and clinicopathological characteristics of the patients included in the study
Characteristics | Level | N | N0 | N1 | P value |
Gender | Male | 56 | 34 | 22 | 0.194 |
| Female | 200 | 141 | 59 | |
Age | < 45 | 109 | 70 | 39 | 0.225 |
| ≥ 45 | 147 | 105 | 42 | |
Maximum tumor diameter(mm) | > 5 | 73 | 36 | 37 | 0.000*** |
| ≤ 5 | 183 | 139 | 44 | |
Number of lesions | Single | 121 | 102 | 19 | 0.000*** |
| Multiple | 135 | 74 | 61 | |
ACR score | >5 | 90 | 31 | 59 | 0.000*** |
| ≤ 5 | 166 | 144 | 22 | |
*** p < 0.001 |
The predictive factors were further analyzed using Multivariate Logistic Regression. Detailed statistical results can be found in Table 2. It was also found that maximum tumor diameter, number of lesions, and ACR score are all independent risk factors for cervical lymph node metastasis in PTMC patients.
Table 2
Statistical results of predictor variables from Multivariate Logistic Regression
Characteristics | Estimate | Std.Error | P value |
Gender | 0.412 | 0.397 | 0.2995 |
Age | -0.029 | 0.016 | 0.0718 |
Maximum tumor diameter(mm) | 0.159 | 0.074 | 0.0307* |
Number of lesions | 0.916 | 0.356 | 0.0101* |
ACR score | 2.349 | 0.344 | 0.0000*** |
* p < 0.05 ** p < 0.01 *** p < 0.001 |
Randon forest model
Given the potential for complex and nonlinear relationships among predictive variables and outcome indicators, to construct the most suitable Randon Forest prediction model, we first use all five predictive factors (age, gender, maximum tumor diameter, number of lesions, and ACR score) to build Model 1. Then, we use independently verified risk factors (maximum tumor diameter, number of lesions, ACR score) to construct Model 2. The weights of each predictive factor in Model 1 and Model 2 are shown in Fig. 2a and 2b, respectively. From this, we can conclude that in Model 1, the maximum tumor diameter contributes the most to the model, while in Model 2, the ACR score has the most enormous contribution. The comparison of model parameters is as follows: Model 1 (AUC = 0.92, optimal threshold = 0.46, sensitivity = 0.83, specificity = 0.90), Model 2 (AUC = 0.89, optimal threshold = 0.28, sensitivity = 0.88, specificity = 0.83)(Table 3). Additionally, we have plotted the ROC curves for both models and annotated their AUC values (Fig. 3a and 3b). By comparing the above parameters and following the principles for selecting the optimal model, Model 1 is identified as the best model. The models mean that including all five predictive factors (age, gender, maximum tumor diameter, number of lesions, ACR score) in the model achieves better predictive performance.
Table 3
Summary table of optimal thresholds, sensitivity, and specificity for four models
| Optimal Threshold | Sensitivity | Specificity | AUC |
Randon Forest | 0.461 | 0.831 | 0.901 | 0.921 |
| 0.282 | 0.882 | 0.832 | 0.892 |
SupportVector Machine | 0.161 | 0.831 | 0.811 | 0.841 |
| 0.182 | 0.872 | 0.832 | 0.872 |
Logistic Regression | 0.211 | 0.901 | 0.741 | 0.871 |
| 0.212 | 0.872 | 0.812 | 0.872 |
Xgboost | 0.461 | 0.721 | 0.891 | 0.831 |
| 0.242 | 0.792 | 0.832 | 0.842 |
1 Model 1 2Model 2 |
Support vector machine
When constructing the Support Vector Machine model, we first use five predictive factors to build Model 1 and then three independent risk factors to construct Model 2. The weights of the variables in the models are shown in Fig. 2c and 2d. It can be observed that the number of lesions(nodes) is the factor contributing most significantly to the model. To visually compare the two models, we plotted the ROC curves of the models and annotated the AUC values (Fig. 1c, 1d). The specific parameters for Model 1 and Model 2 are as follows: Model 1 (AUC = 0.84, optimal threshold = 0.16, sensitivity = 0.83, specificity = 0.81); Model 2 (AUC = 0.87, optimal threshold = 0.18, sensitivity = 0.87, specificity = 0.83) (Table 3). According to the principles of selecting the optimal model, Model 2 is the superior Support Vector Machine model.
Multivariate logistic regression
Although our analysis of the collected data from all 256 patients suggests that age and gender are not independent risk factors, it is vital to consider domain-specific expertise when building models. Previous research has indicated that age and gender are independent risk factors for cervical lymph node metastasis in patients with PTMC[30, 39]. Therefore, our study also constructed two models using multinomial Logistic regression: Model 1, which includes all five predictive factors, and Model 2, which provides only three independent risk factors. Based on the bar charts of the weights of various predictive factors (Fig. 4a and 4b), the ACR score is the most crucial variable in the Multivariate Logistic Regression model. The ROC curves for Model 1 and Model 2, along with their respective AUC values, can be seen in Fig. 5a and Fig. 5b. The specific parameters for the models are as follows: Model 1 (AUC = 0.87, optimal threshold = 0.21, sensitivity = 0.90, specificity = 0.74); Model 2 (AUC = 0.87, optimal threshold = 0.21, sensitivity = 0.87, specificity = 0.81) (Table 3). Both models have the same AUC value; however, Model 1 has higher sensitivity than Model 2, making it the better multivariate logistic model.
Xgboost model
Using the Xgboost model, a robust machine learning model capable of handling complex nonlinear relationships between factors, we constructed two models: Model 1 using all five predictive factors and Model 2 using only three independent risk factors. The maximum tumor diameter is the factor with the most excellent weight in both models (Fig. 4c and 4d). The ROC curves and AUC values for both models are also presented in Fig. 5c and 5d. The specific parameters are as follows: Model 1 (AUC = 0.83, optimal threshold = 0.46, sensitivity = 0.72, specificity = 0.89); Model 2 (AUC = 0.84, optimal threshold = 0.24, sensitivity = 0.79, specificity = 0.83) (Table 3). Model 2 is considered the better Xgboost model with a higher AUC value and sensitivity.
Construction of the best model and predictive website
Based on the information provided and by comparing the parameters of each model, as shown in Table 3, we ultimately selected Randon Forest Model 1 as the best predictive model. To further validate the model's stability, we conducted a ten-fold cross-validation. The AUC values for the ten validations were as follows: 0.838, 0.910, 0.865, 0.873, 0.863, 0.872, 0.849, 0.825, 0.867, and 0.941, with an average AUC value of 0.870. A curve graph was also plotted to visually demonstrate this (Fig. 6). From the results, it can be observed that the model is highly stable, with the AUC values maintained above 0.80 in 10 validations. Based on this model, we have developed a website to allow clinicians to access the prediction results easily. The website's webpage is shown in Fig. 7, and the link to the website is as follows: http://yucemoxing.online:8082.
Construction of the online platform
To enhance the generalization ability of our model and allow it to train on more data, we have provided online open platforms that support users in uploading their data for model building, selection, and application. Since different data might yield varying performance outcomes with the models, we offer researchers ample choice to select the optimal model. We provide eight different model-building platforms, based respectively on two variations each of the Randon Forest model(Model1, Model2), Support Vector Machine model(Model 1, Model 2), multinomial Logistic Regression model(Model 1, Model 2), and the Xgboost model(Model 1, Model 2). The interfaces of the online platforms are shown in Fig. 8 and Fig. 9, and the corresponding links to the platforms are as follows:
http://yucemoxing.online:8081(Randon Forest Model 1);
http://yucemoxing.site:8089 (Randon Forest Model 2);
http://yucemoxing.online:8084( Support Vector Machine Model 1) ;
http://yucemoxing.online:8085( Support Vector Machine Model 2) ;
http://yucemoxing.online:8083( Multivariate Logistic Regression Model 1);
http://yucemoxing.online:8088( Multivariate Logistic Regression Model 2);
http://yucemoxing.online:8087(Xgboost Model 1);
http://yucemoxing.online:8086(Xgboost Model 2) .